jesamjasem,

You might want to consider which window has focus for the key listener, I'd try making the JPanel request focus :)

whilst you have set it to focusable you may need to request it
...