Combine flour, salt, cinnamon and cardamom in a large bowl. Add butter and cut it into flour with a pastry blender or two knives until pieces are the size of peas. Add granulated sugar, oats, pecans, ginger and orange zest and use your fingertips to combine until topping is crumbly. Store in an airtight container in the refrigerator until you are ready to use it, or up to 2 days.
To peel peaches, bring a large pot of water to a boil. Create an ice bath: Place 4 to 6 cups ice in a large bowl. Add enough cold tap water to almost fill bowl.
Cut an X through skin at top of each peach. Plunge peaches into boiling water for about 1 to 2 minutes, until skin begins to curl up at cut. Drain peaches and plunge them into ice bath. Starting at cut where skin is already rolling back, peel peaches; discard skin. Cut peaches into 1/2-inch wedges so they fall into a large bowl.
Heat oven to 350°. Stir together brown sugar, granulated sugar (if using), cornstarch and nutmeg; sprinkle over peaches and toss gently to coat. Pour peaches and any juice in bowl into a baking dish. Scatter topping over fruit. Put baking dish on a baking sheet to catch any juice that may boil over. Bake at 350° for 50 to 60 minutes, until top is golden and fruit is bubbling.
Cool slightly before serving warm with scoops of vanilla ice cream.