Your problem is that you are doing this with a TCP connection. TCP connections are guaranteed to pass data successfully from A to B and allow a one way transfer of data at any one time (unless you have two TCP connections). If you want to get around this, create UDP connections and use Datagram Packets. They don't require feedback from the client (as a rule in networking, the client is the computer receiving the data, whether it's actually the server or not) and won't hang if the client does not receive the packet, they just keep on transmitting.
You send data out through UDP as a stream of bytes which you can then turn back into any object you wish on the other side.