Remove the mouse listener, or add a conditional in the listener method that checks if the button is enabled.