Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 4 of 4

Thread: Bulk write of big int[] and long[]

  1. #1
    Junior Member
    Join Date
    Dec 2011
    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Angry Bulk write of big int[] and long[]

    Situation:
    I have an int[60 million] and a long [240 million] and i want to save them to a file.

    Problem:
    I need a bulk write like FileOutputStream.write(byte[] b, int off, int len).
    All bulk writes seem to require byte [] or ByteBuffer.
    There seems to be no way to convert an int[] to a byte[] or ByteBuffer.

    Failed attempts:
    I tried allocating my arrays with ByteBuffer.allocate() and then using ByteBuffer.asLongBuffer.array(), but that function is not supported.
    I tried staying with a LongBuffer instead, however, LongBuffer.put() is also optional and not supported.
    I tried to use ByteBuffer.wrap(new byte[]).asLongBuffer.array(), but the function is still not supported.
    Converting my longs to byte[] via bitshifting will take ages, and it is a time-critical application.

    Possible solutions:
    Find a bulk write that accepts int[] and long[].
    Find an intermediate type that can be backed by int[] and long[] which is supported by a bulk write.

    Note:
    I come from C and am very frustrated to not be able to just say FileOutputStream.write((byte [])int[]).
    Yes, I used google and could not find anything useful.
    Any help would be greatly appreciated.

    Edit:
    Im on Win7x64
    java version "1.6.0_29"
    Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
    Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
    Last edited by nwp; December 4th, 2011 at 03:01 PM. Reason: additional platform information


  2. #2
    Administrator copeg's Avatar
    Join Date
    Oct 2009
    Location
    US
    Posts
    5,320
    Thanks
    181
    Thanked 833 Times in 772 Posts
    Blog Entries
    5

    Default Re: Bulk write of big int[] and long[]

    Why do you need a bulk write? You can use a DataOutputStream, loop over the array and write the appropriate data type (to read you would use a DataIntputStream).

    however, LongBuffer.put() is also optional and not supported
    What do you mean by this? LongBuffer is abstract, and any instantiation of the class (eg through allocate) must implement the method

  3. #3
    Super Moderator helloworld922's Avatar
    Join Date
    Jun 2009
    Posts
    2,896
    Thanks
    23
    Thanked 619 Times in 561 Posts
    Blog Entries
    18

    Default Re: Bulk write of big int[] and long[]

    Consider wrapping your FileOutputStream using a DataOutputStream and use the writeInt() and writeLong() methods. To speed up writes, it may also be beneficial to put an intermediary BufferedOutputStream (you'll have to benchmark it yourself to see what the effects are).


  4. The Following User Says Thank You to helloworld922 For This Useful Post:

    nwp (December 4th, 2011)

  5. #4
    Junior Member
    Join Date
    Dec 2011
    Posts
    2
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default Re: Bulk write of big int[] and long[]

    Quote Originally Posted by helloworld922 View Post
    Consider wrapping your FileOutputStream using a DataOutputStream and use the writeInt() and writeLong() methods. To speed up writes, it may also be beneficial to put an intermediary BufferedOutputStream (you'll have to benchmark it yourself to see what the effects are).

    I tried DataOutputStream out = new DataOutputStream(new FileOutputStream(some_file)) but gave up after a few minutes.
    With BufferedOutputStream in the middle it takes 140 seconds, and considering the number of IO-operations I doubt the IO is actually buffered.
    I know it has been done in the realm of 10 seconds, but i will take it for now. Thanks.

Similar Threads

  1. Converting Object to Long
    By aussiemcgr in forum Java Theory & Questions
    Replies: 5
    Last Post: August 20th, 2010, 08:35 AM
  2. Sudoku Creator (this could be a long one)
    By aussiemcgr in forum Java Theory & Questions
    Replies: 8
    Last Post: July 27th, 2010, 12:19 PM
  3. How long did it take you to learn?
    By JavaLearner in forum The Cafe
    Replies: 5
    Last Post: March 24th, 2010, 04:42 PM
  4. Bulk operations on sets/ map problem
    By kyuss in forum What's Wrong With My Code?
    Replies: 0
    Last Post: March 14th, 2010, 12:48 PM
  5. The input line is too long
    By vivekmk in forum Java IDEs
    Replies: 5
    Last Post: October 16th, 2009, 10:35 PM

Tags for this Thread