Java program runs out of memory (-xms1024 -xmx1024)

I do not know why my java program runs out of memeory about half way through the list?

Code java:

---

public static void getMovieFile() {
 
        int myBuffer, counter = 0;
        String aFile = "movies.list";
        Pattern myIgnore = Pattern.compile("\".*");
        Pattern myYear = Pattern.compile("\\(\\d\\d\\d\\d.*");
        String title, year;
        Scanner myScanner;
 
 
        try {
            //New Scanner stream for the file input (aFile)
            myScanner = new Scanner(new FileInputStream(aFile), "ISO-8859-1");
 
            //Skip first 20 lines (Movies are at line 1,312,533)
            while(counter<20){
                myScanner.nextLine();
                counter++;
            }
            //Reset counter
            counter = 0;
            //While Scanner starts with quotation marks (") skip line
            while (myScanner.hasNext(myIgnore)){
                myScanner.nextLine();
            }
 
            //While Scanner has next token read token
            while (myScanner.hasNext()) {
                //Declaring and instantiating title (Blank)
                title = "";
                //While Scanner is not movie year concat title with next token
                while(!myScanner.hasNext(myYear)){
                    title = title.concat(myScanner.next()).concat(" ");
 
                }
                //Else return and substring movie year
                year = myScanner.next().toString().substring(1,5);
                //Inputting title and year into the object database
                //While year is not empty
                while(!(year.isEmpty()&&title.isEmpty())){
                    try{
                        //Check if year is a valid integer
                        myBuffer = Integer.parseInt(year);
                        //Input records into array
                        MovieMenu.movies.inputMovie(title, myBuffer, "NA", "NA", 0.0, "NA", "NA", counter);
                        //Reset title and year for new input
                        System.out.println("Inputting Movie: " +title +"Year:" +year +"Position: " +counter);
                        title = year = "";
                    } catch(NumberFormatException e) {
                        //Parse Integer Number format exception
                        Logger.getLogger(MovieListImport.class.getName()).log(Level.SEVERE, null, e);
                        System.out.println("Error: Invalid release");
                    }
                    //Increase position in movies array list
                    counter++;
                }
                //Move Scanner to next line
                myScanner.nextLine();
            }
            //Close myScanner
            myScanner.close();
 
        } catch (FileNotFoundException ex) {
            //FileInputStream File not found exception
            Logger.getLogger(MovieListImport.class.getName()).log(Level.SEVERE, null, ex);
            System.out.println("Error: File not Found");
        } catch (SecurityException e){
            //FileInputStream Security exception
            Logger.getLogger(MovieListImport.class.getName()).log(Level.SEVERE, null, e);
            System.out.println("Error: Cannot read file");
        } catch (NullPointerException ioe){
            //Scanner Null pointer exception
            Logger.getLogger(MovieListImport.class.getName()).log(Level.SEVERE, null, ioe);
            System.out.println("Error: Corrupt file");
        } catch (NoSuchElementException e){
            //Scanner No such element exception
            Logger.getLogger(MovieListImport.class.getName()).log(Level.SEVERE, null, e);
            System.out.println("Error: End of file");
        }
 
    }

---

Is one of your loops out of control? Add some printlns in the loops to see if any of them are not stopping.

It does appear that all my while loops do finish when they are meant to on a short text file: Although the files it is trying to read in are 100mb+ lists from IMDB. When running through the IMDB files it gets to about a quater of the way through before crashing. When I remove all System.out.println in my method the concat does not work and it just reaches end of file error after the first line. Should I be using concat or is that not the problem? If I created a new scanner or stringbuffer when I wanted to concat / append the title would that stop the crashing?

Code :

---

Skipping Line: 1
While loop (Skip) finished
Counter reset to (0): 0
Skipping Line if starting with ("): 
While loop (Ignore) finished:
myScanner.next(): #1
concat of myScanner.next(): #1 Movie
concat of myScanner.next(): #1 Movie Test
concat of myScanner.next(): #1 Movie Test File
While loop (Title) finished: #1 Movie Test File
 
Inputting Movie:
	Title: #1 Movie Test File
	Year: 2005
	Position: 0
While loop (Input) finished:
 
While loop (hasNext()) finished:

---

What do you wish to do with the information afterwards? In other words, do you need to store all the information in memory? My guess is probably not, in which case you may wish to parse the files into some type of relational database that can be more readily accessed at runtime (100mb isn't that large, but loading it all (or many of them) into memory adds quite a bit of overhead). FWIW, I would recommend a StringBuilder for your task (as written this should not impact your out of memory exception however)

100MB is pretty large for text, but try appending to a StringBuilder, not buffer as concatenating is a costly business.

Edit: Seems I got beat to it :p