I used struts, and preferred sizes to force it to do that. However I decided to use the BoxLayout, because it does what I need it to do, I was able to fix the problem I was having. What I did to solve it was, I changed the frame to a BoxLayout Y-Axis, and added 3 panels that use the BoxLayout X-Axis, I then changed the size of the panels to push the buttons closer together, to make it look nicer. That seemed to do the trick, if I drag it out, it has the result of the bottom image of my original post.
Anyway thanks for attempting to help.