Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 6 of 6

Thread: Out of memory work around for a java application (please help!)

  1. #1
    Junior Member
    Join Date
    Jan 2010
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Question Out of memory work around for a java application (please help!)

    I am working on a java application(Batch Process) and we have to use some third party API's (No Option to change it) related to the Database.It is causing out of memory after every 25000 updates and the total records are 5 million or so. I used the JProfiler to find out that the out of memory is caused by the third party API only.Since the application is in the production already we really cannot risk to change that third party API at any cost so we have to do some work around.

    What is the best work around in this scenario? I am thinking to create separate thread in the below mentioned code which will do the first 20000 updates and as soon as this thread dies all the objects associated with this will also die and hence the new thread will be a fresh start and again the new thread will do 20000 updates or something like that.

    If you have encountered such kind of problem in the past please help me and if possible write a pseudo code or sample code for it.

    my current code looks like this which I am thinking to add the java threads:

    public static void main(String [] args){

    *ResultSet* rs (rs has all the records e.g. 5 millions)
    long count=0;
    while (rs.next()){

    doUpdate();

    count++;
    *//the code fails if the count reaches to 25000*

    }


    }


  2. #2
    Administrator copeg's Avatar
    Join Date
    Oct 2009
    Location
    US
    Posts
    5,267
    Thanks
    176
    Thanked 821 Times in 764 Posts
    Blog Entries
    5

    Default Re: Out of memory work around for a java application (please help!)

    Have you tried to increase the JVM default heap size? I believe the default Heap size is 128mb, but you can increase the max size into the GB if needed (using the -Xmx(max mem size) command line argument). However this doesn't fix any underlying problem. If you are creating new objects in the while loop and keeping references to them, then you might want to consider writing them out to a file rather than keeping a reference (if at all possible).

  3. #3
    Junior Member
    Join Date
    Jan 2010
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Re: Out of memory work around for a java application (please help!)

    Thanks for your quick reply. I have already tried to use max and min memory setting, its not working. The other thing is The third party API does all the stuff and they only have provided .jar files so really I cannot change anything(e.g. writing the obj into the file etc.). So the only option I have is somehow I have to start the process again with the new records in the same JVM. So that I can have a fresh start. I dont know how to achieve this.

  4. #4
    Administrator copeg's Avatar
    Join Date
    Oct 2009
    Location
    US
    Posts
    5,267
    Thanks
    176
    Thanked 821 Times in 764 Posts
    Blog Entries
    5

    Default Re: Out of memory work around for a java application (please help!)

    Not an easy problem to debug without knowing more. Like I mentioned make sure you aren't keeping references to any objects, and try not to re-use objects if you are creating them from the external jar.
    You could try the thread process, but I'm guessing you may end up in the same boat, however arranging it that way you may stumble upon the problem and be able to overcome it. Below is a serious hack and NOT something I'd want to place in distributed software but if this is a one time deal/script and it'd faster to just get it done than messing around with all the internals might be worth a try. (I'd much rather fix the underlying problem). The idea is you could iteratively go through your main loop by sending in your count as the argument...something along these lines:

     
    public static void main(String [] args){
    long count=0;
    if ( args.length != 0 ){
        count = Integer.parseInt(args[0]);
    }
    *ResultSet* rs (rs has all the records e.g. 5 millions)
    int index = 0;
    if ( count != index ){//skip these
        while ( rs.next() ){
             index++;
            if ( index == count ){
                break;
            }
        }
    }
     
    while (rs.next()){
     
    doUpdate();
     
    count++;
    if ( count != 0 && count % 20000 == 0 ){//only do 20000 at a time, the restart the JVM at this count
        Runtime.getRuntime().exec("java YourAppName " + Integer.toString(count));
        System.exit(0);
    }
    *//the code fails if the count reaches to 25000*
     
    }
     
     
    }

    Again, a warning this is a VERY MESSY hack - use at your own risk. This may not fix the problem, for all I know its the rs.next() call that's the problem
    Last edited by copeg; January 20th, 2010 at 02:46 PM.

  5. #5
    Junior Member
    Join Date
    Jan 2010
    Posts
    7
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default Re: Out of memory work around for a java application (please help!)

    Are you getting the data from the database and loading the ResultSet? If so, you can change how you load the data and batch the resultsets... but if the 3rd party API is creating the resultset... then you might have to go with the hack...
    One other thing... is the 3rd part a class you pass a record at a time and only at the 25000 record it hoses? If that the case load/unload the 3rd party class every 20000 records...
    Hope this helps!
    ------------------------------
    Mike

  6. #6
    Super Moderator Json's Avatar
    Join Date
    Jul 2009
    Location
    Warrington, United Kingdom
    Posts
    1,274
    My Mood
    Happy
    Thanks
    70
    Thanked 156 Times in 152 Posts

    Default Re: Out of memory work around for a java application (please help!)

    Are you closing your database properly? This sounds like a leak to me.

    // Json

Similar Threads

  1. Replies: 0
    Last Post: December 3rd, 2009, 03:43 PM
  2. Out of memory - Java heap space
    By fraxx in forum Java Theory & Questions
    Replies: 4
    Last Post: November 24th, 2009, 04:26 AM
  3. Help with Java Application
    By Riston in forum File I/O & Other I/O Streams
    Replies: 1
    Last Post: November 19th, 2009, 06:17 PM
  4. java application connecting to a server
    By wildheart25c in forum Java Networking
    Replies: 2
    Last Post: September 17th, 2009, 07:22 AM
  5. Java memory management
    By trueacumen in forum Java Theory & Questions
    Replies: 5
    Last Post: August 12th, 2009, 02:40 AM