Bulk write of big int[] and long[]
Situation:
I have an int[60 million] and a long [240 million] and i want to save them to a file.
Problem:
I need a bulk write like FileOutputStream.write(byte[] b, int off, int len).
All bulk writes seem to require byte [] or ByteBuffer.
There seems to be no way to convert an int[] to a byte[] or ByteBuffer.
Failed attempts:
I tried allocating my arrays with ByteBuffer.allocate() and then using ByteBuffer.asLongBuffer.array(), but that function is not supported.
I tried staying with a LongBuffer instead, however, LongBuffer.put() is also optional and not supported.
I tried to use ByteBuffer.wrap(new byte[]).asLongBuffer.array(), but the function is still not supported.
Converting my longs to byte[] via bitshifting will take ages, and it is a time-critical application.
Possible solutions:
Find a bulk write that accepts int[] and long[].
Find an intermediate type that can be backed by int[] and long[] which is supported by a bulk write.
Note:
I come from C and am very frustrated to not be able to just say FileOutputStream.write((byte [])int[]).
Yes, I used google and could not find anything useful.
Any help would be greatly appreciated.
Edit:
Im on Win7x64
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
Re: Bulk write of big int[] and long[]
Why do you need a bulk write? You can use a DataOutputStream, loop over the array and write the appropriate data type (to read you would use a DataIntputStream).
Quote:
however, LongBuffer.put() is also optional and not supported
What do you mean by this? LongBuffer is abstract, and any instantiation of the class (eg through allocate) must implement the method
Re: Bulk write of big int[] and long[]
Consider wrapping your FileOutputStream using a DataOutputStream and use the writeInt() and writeLong() methods. To speed up writes, it may also be beneficial to put an intermediary BufferedOutputStream (you'll have to benchmark it yourself to see what the effects are).
Re: Bulk write of big int[] and long[]
Quote:
Originally Posted by
helloworld922
Consider wrapping your FileOutputStream using a
DataOutputStream and use the writeInt() and writeLong() methods. To speed up writes, it may also be beneficial to put an intermediary BufferedOutputStream (you'll have to benchmark it yourself to see what the effects are).
I tried DataOutputStream out = new DataOutputStream(new FileOutputStream(some_file)) but gave up after a few minutes.
With BufferedOutputStream in the middle it takes 140 seconds, and considering the number of IO-operations I doubt the IO is actually buffered.
I know it has been done in the realm of 10 seconds, but i will take it for now. Thanks.
