Literals aren't limited to only numbers. They can be booleans, characters, strings, numbers, null, etc. (not a complete list).
They're called literals because they represent an absolute value which cannot be interpreted as anything other else. We call them literals because we need a name to refer to what these things are.
There is a big difference between these:
int val = '4'; // not actually the integer literal 4, but the character literal '4' which in ASCII is equivalent to 52
int val2 = 4; // this is the integer literal 4
int val3 = (int) 4f; // this is the floating point literal 4f. Because floats are not implicitly castable to int, we must explicitly cast it.
There are complicated reasons (some of which is historical) why integer literals and long literals are treated differently, and I won't pretend to know/understand all of it. My guess is that it has something to do with implementation details of the compiler (especially on older 32-bit systems), and possibly has historical reasons coming from C/C++ where ints originally held 16-bit values and longs held 32-bit values.