Cover potatoes in a large pot with lightly salted water. Bring to a boil; cook until tender enough to easily pierce with a fork, about 20 minutes.
While potatoes cook, heat butter and milk in a small saucepan over low heat until very warm.
Drain and partially mash potatoes. Add butter and milk mixture, then continue mashing until smooth. Stir in goat cheese, chives, and dill. Add more milk as needed to reach desired consistency. Season to taste with salt and pepper.
Hold back a cup of the plain mashed potatoes for the kids. Then add the goat cheese, dill, and chives to the rest for the grown-ups.