Preheat oven to 375 F. Line 36 mini muffin cups. Place plums and buttermilk in a food processor; pulse until smooth. Combine flours, cinnamon, baking powder, baking soda, ginger, salt, and nutmeg in a small bowl. Beat 1/4 cup butter and the brown sugar in a large bowl. Add egg and beat well. Blend in flour mixture in thirds, alternating with buttermilk mixture, ending with flour.
Fill muffin cups. Bake until a toothpick inserted into center comes out clean, 10 to 12 minutes. Cool on a wire rack.
For the frosting, beat cream cheese and remaining 1/2 cup butter in a bowl until smooth. Gradually add the confectioners' sugar. Tint with food coloring. Spoon into a pastry bag fitted with a #113 leaf pastry tip. Starting on the outer edge of cupcakes, pipe lines of frosting to look like petals on a tutu. Continue with a second row, leaving center of the cupcakes open. Sprinkle with glitter.