Preheat the oven to 400. Prepare a regular sized muffin tin with muffin liners or grease. Set aside.
In a large bowl, rub together the sugar and lemon zest with a fork until the sugar and zest are one. Then, whisk in the flour, baking powder, baking soda, and salt.
In a large bowl, whisk together the sour cream, yogurt, eggs, vanilla, lemon juice, and melted butter. Pour the combined ingredients into the dry ingredients and mix fully. Do not over mix the batter. It is okay if there are a few lumps in the batter.
Divide the batter evenly among the muffin tin and bake for 15-18 minutes. The muffins will be done when the tops are golden brown or a toothpick or cake tester comes out clean.
Let the muffins cool on a wired rack for about 10 minutes. While the muffins are cooling, make the glaze. Combine the confectioner's sugar, milk, and extract in a bowl and stir until smooth. Drizzle the glaze over the lemons. You may need to add more or less milk to get the desired consistency of the glaze.
Seal in an airtight container for storage.