Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 4 of 4

Thread: Reading In Then Writing Out Massive Files

  1. #1
    Forum VIP
    Join Date
    Jul 2010
    Posts
    1,676
    Thanks
    25
    Thanked 329 Times in 305 Posts

    Default Reading In Then Writing Out Massive Files

    Ok, so I have a MASSIVE text file (2,148,327 characters). I need to read in the text file, edit some of the data, then write it back out (not necessarily in the same order that it was read in).

    This procedure will eventually be implemented in a web program that I have on a free host server. For the time being, I am figuring out how to do this on my local machine.

    At the moment, I am reading in data, storing the data I need to edit in an object vector and storing the rest of the data in a String, then writing the data back out later. The problem I am having is that I am running into memory walls while creating the massive String (thanks to the Arrays.copy() method that the String class uses to resize Strings). In an ideal world, I could use a database for this but, as far as I know, my host does not support applets to communicate with databases. Increasing the heap stack size will not be a solution because the same would need to be done on the server (or clients perhaps), which I cannot do. So, I need to figure out a better way to read and write the data.

    Due to it being web based, I also have to communicate with the files via Streams. Theoretically, I could backup the input file, erase the original file, read input from the backup, store the data I need to edit in my object vector and append the rest of the data to the original file, and then append the edited data to the file at the end. That way I wouldn't be storing the excess data in memory. The problem with that is I don't think I can append onto Streams or, if I can, I have no idea how.

    Does anyone have any suggestions as to what I should do here? I've got no clue what to do.
    NOTE TO NEW PEOPLE LOOKING FOR HELP ON FORUM:

    When asking for help, please follow these guidelines to receive better and more prompt help:
    1. Put your code in Java Tags. To do this, put [highlight=java] before your code and [/highlight] after your code.
    2. Give full details of errors and provide us with as much information about the situation as possible.
    3. Give us an example of what the output should look like when done correctly.

    Join the Airline Management Simulation Game to manage your own airline against other users in a virtual recreation of the United States Airline Industry. For more details, visit: http://airlinegame.orgfree.com/


  2. #2
    Forum old-timer
    Join Date
    Nov 2008
    Location
    Faversham, Kent, UK
    Posts
    472
    My Mood
    Mellow
    Thanks
    4
    Thanked 58 Times in 54 Posts

    Default Re: Reading In Then Writing Out Massive Files

    You can open a FileOutputStream for appending by using the constructor that takes an 'append' indicator.

  3. #3
    Forum VIP
    Join Date
    Jul 2010
    Posts
    1,676
    Thanks
    25
    Thanked 329 Times in 305 Posts

    Default Re: Reading In Then Writing Out Massive Files

    In order to append, does it read in the data the file already contains? Because if it does, I'll have the same problem.

    EDIT: This actually seems to work fantastically on my computer. In a perfect world I would be able to add to the beginning to the file instead of the end, but that is more of a convenience instead of a requirement. We'll see how it works on the server when I try it on there in a few days.
    Last edited by aussiemcgr; July 28th, 2011 at 03:19 PM.
    NOTE TO NEW PEOPLE LOOKING FOR HELP ON FORUM:

    When asking for help, please follow these guidelines to receive better and more prompt help:
    1. Put your code in Java Tags. To do this, put [highlight=java] before your code and [/highlight] after your code.
    2. Give full details of errors and provide us with as much information about the situation as possible.
    3. Give us an example of what the output should look like when done correctly.

    Join the Airline Management Simulation Game to manage your own airline against other users in a virtual recreation of the United States Airline Industry. For more details, visit: http://airlinegame.orgfree.com/

  4. #4
    Forum old-timer
    Join Date
    Nov 2008
    Location
    Faversham, Kent, UK
    Posts
    472
    My Mood
    Mellow
    Thanks
    4
    Thanked 58 Times in 54 Posts

    Default Re: Reading In Then Writing Out Massive Files

    Quote Originally Posted by aussiemcgr View Post
    In order to append, does it read in the data the file already contains?
    No, it's an OutputStream, so it doesn't read data in. It basically relies on the OS to sort out the details.

Similar Threads

  1. Writing to files
    By nitwit3 in forum Java Theory & Questions
    Replies: 3
    Last Post: July 25th, 2011, 04:00 AM
  2. file reading & writing
    By macko in forum What's Wrong With My Code?
    Replies: 7
    Last Post: May 12th, 2011, 08:54 AM
  3. Reading a writing binary file
    By stu1811 in forum File I/O & Other I/O Streams
    Replies: 1
    Last Post: July 30th, 2010, 12:58 PM
  4. Reading and Writing Text Files
    By kappasig84 in forum File I/O & Other I/O Streams
    Replies: 1
    Last Post: March 1st, 2010, 07:16 PM
  5. Reading many files using a scanner
    By jayjames90 in forum File I/O & Other I/O Streams
    Replies: 2
    Last Post: October 22nd, 2009, 04:35 PM