Cook pudding with milk according to package directions, 4 to 6 minutes. Cool 30 minutes in refrigerator; stir in yogurt.
In a 14- to 16-cup footed glass trifle dish, layer 2 cups of the cake cubes, one-third of the pudding mixture, one-third of the mangoes and one-third of the raspberries. Repeat layering twice. Cover and refrigerate overnight.
Whip cream, sugar and vanilla until medium-soft peaks form. Swirl over top of trifle. Top with almonds.