The best way to translate the code is to figure out fundamentally what the intent of the code is.
ulong num = 0uL;
for (int j = 0; j < 5; j++)
num <<= 8;
num |= (ulong)keyHex[i * 5 + j];
This code takes each byte value from keyHex and shifts it into the LSB of num. There are 5 bytes shifted in.
for (int j = 0; j < 8; j++)
ulong num2 = num >> j * 5 & 31uL;
char c = text[(int)num2];
arg += c;
This selects 5 bit chunks from num, working from the LSB to the MSB. There are 8 5-bit chunks. It then maps that value to the appropriate character and appends it to the output string.
So the sole purpose of this code is to take 5-bit compressed values and expand them out, then map the results appropriately.
The question then becomes how can you replicate this behavior in Java.
As it turns out, pretty much the same exact code will work in Java (with a few minor differences to account for library and language semantics). Why? Because you never use the msb of the unsigned long, so it doesn't matter that it's unsigned.