In a medium bowl stir together cocoa powder and sugar. Stir in water to make a paste. Whisk in milk until smooth. Pour mixture into 9 3-ounce pop molds. Insert plastic pop sticks or cover with foil and make slits for wooden sticks. Freeze until completely solid, 3 to 4 hours.