One thought would be to change the contents of the JButtons instead of moving them around.