Preheat your oven to 350 degrees F. Grease and flour the inside of the cake pans, being sure to tap out the excess flour. Set aside.
In a medium mixing bowl, stir together the flour, baking powder, baking soda, and salt. Set aside.
In the bowl of a stand mixer (or a large mixing bowl with a hand mixer), beat together butter and sugar until light and fluffy, about 2 minutes. Reduce the mixer speed to low and add in eggs and egg yolks, one at a time. Mix in the lemon juice and zest.
Alternate adding the flour mixture and buttermilk to the cake batter, beginning and ending with the flour. Mix until just combined, scraping the bowl as needed.
Divide the cake batter evenly between the two prepared pans. Place them in the oven and bake for 30-35 minutes. The cakes are done when they start to pull away from the sides of the pan and a toothpick inserted in the middle comes out clean.
Remove the cakes from the oven and cool in the pans on a wire rack for 10 minutes on a wire rack. Then remove the cakes from the pans and cool completely on the wire racks. (Warm cakes will melt the frosting and you don't want that!)
Lime Buttercream
Once the cakes are cooled, make the buttercream. In a large bowl, beat together the powdered sugar and butter until smooth. Add the lime zest and juice and mix to combine.
Pour in 1/2 of the milk and beat until incorporated. Pour in more milk as needed until the frosting is smooth but still stiff. Beat for about 5 minutes until fluffy. (You can tint the frosting with green food coloring if desired.)
Assembly
Place one of the cake layers on a plate or cake stand, bottom side down. Spread about 1/3 of the frosting over the cake all the way to the edges. Place the second cake layer over the first. (If you want a flat top, turn the top layer bottom-side up.) Spread the rest of the frosting over the top and sides of the cake. To garnish, arrange fresh or candied lemon and lime slices over the top of the cake. Cut into 10 slices before serving.
Notes
If you don't have buttermilk on hand you can make sour milk instead. Mix together 1 cup of milk with 1 tablespoon of lemon juice or white vinegar. Stir and let sit for 5 minutes before using.