I recommend nesting panels in panels, like newbie suggested. The trick is to figure out what layout managers to put in each.
Check out this: A Visual Guide to Layout Managers
If you are having trouble doing this in code, you might want to try a GUI based builder. NetBeans is a free IDE written by Sun for writing java and it has a GUI tool to help you layout components, like Visual Basic sort of. Eclipse may have something similar now - I am not sure.
Sometimes I cheat and use NetBeans, and look at the resulting code to see how they did it, and then write my own using that info.