Welcome to the Java Programming Forums


The professional, friendly Java community. 21,500 members and growing!


The Java Programming Forums are a community of Java programmers from all around the World. Our members have a wide range of skills and they all have one thing in common: A passion to learn and code Java. We invite beginner Java programmers right through to Java professionals to post here and share your knowledge. Become a part of the community, help others, expand your knowledge of Java and enjoy talking with like minded people. Registration is quick and best of all free. We look forward to meeting you.


>> REGISTER NOW TO START POSTING


Members have full access to the forums. Advertisements are removed for registered users.

Results 1 to 4 of 4

Thread: Loading file into memory is taking up TOO much memory!

  1. #1
    Junior Member
    Join Date
    Mar 2013
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Loading file into memory is taking up TOO much memory!

    Hi, I am attempting to load a small file (Excel format for now) into memory. The excel file is roughly 80 KB in size and only contains about 20 or 30 columns and 80 rows. I've set my JVM parameters to 2 GB. Considering its only an 80 KB file I wouldn't except it to take up more than a MB of memory but for some reason it seems to expand my JVM by 40 MB once loaded!

    Upon startup of my program 40 MB is used according to the task manager, which I think is fine. But for it to increase form 40 to 80 MB on an 80 KB file is too much because I plan on loading much larger files and if memory requirements grow linear then its simply not going to be possible.

    Some classes
    /*
     TableTabInfo Class
     *
     * This Class is a container for all the information that needs to be
     * segregated between multiple loaded tables
     */
    package dataStorageClasses;
     
    import java.util.ArrayList;
    import javax.swing.JTable;
     
    public class [B]TableTabInfo [/B]{
     
        public JTable jDataTable;
        public ArrayList<String> headers;
        public RowData[] rData;
     
        public TableTabInfo() {
            this.headers = new ArrayList<String>();
        }
     
        public void setSizeOfRowData(int size) {
            this.rData = new RowData[size];
     
            for (int i = 0; i < size; i++) {
                this.rData[i] = new RowData();
            }
     
        }
     
        public void addCell(Object cell, int rowNumber) {
            this.rData[rowNumber].addToMyList(cell);
        }
     
        public Object getCell(int row, int col) {
            return this.rData[row].myList.get(col);
        }
     
        public void addToHeaderList(String header) {
            this.headers.add(header);
        }
     
        public void emptyHeaderList() {
            this.headers.clear();
        }
     
        public int sizeOfRowData() {
            return this.rData.length;
        }
    }
     
    package dataStorageClasses;
    import java.util.ArrayList;
     
    public class [B]RowData[/B]<T> {
     
        public ArrayList<T> myList = new ArrayList<T>();
     
        public RowData()
        {
        }
     
        public T getFromMyList(int index) {
            return this.myList.get(index);
        }
     
        public void addToMyList(T e) {
            this.myList.add(e);
        }
    }

    Loading the file
    if (getExtension.contains("xls") || getExtension.contains("lsx")) {
                            try {
                                File inputFile = new File(chooser.getSelectedFile().getAbsolutePath());
                                FileInputStream inputStream = new FileInputStream(inputFile);
     
                                Workbook wb = null;
                                try { 
                                    wb = WorkbookFactory.create(inputStream);
                                } catch (InvalidFormatException ex) {
                                    System.out.println(ex.toString());
                                }
     
                                Sheet sheet = wb.getSheetAt(0);
     
                                //Add a new tab info object into our vector
                                TableTabInfo newTab = new TableTabInfo();
                                newTab.setSizeOfRowData(sheet.getLastRowNum());
     
                                vTabInfo.add(newTab);
     
                                Boolean boolIterOnce = false; 
                                int maxSize = 0; // the variable that is used as the pseudo max
     
                                // Iterate over each row in the sheet
                                Iterator<Row> rowsIterator = sheet.rowIterator();
                                while (rowsIterator.hasNext()) {
                                    Row row = rowsIterator.next();
     
                                    if (!boolIterOnce) //only 1 time for headers
                                    {
                                        Iterator<Cell> cellsIterator = row.cellIterator();
                                        ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addToHeaderList("Row");
     
                                        while (cellsIterator.hasNext()) {
                                            Cell cell = cellsIterator.next();
     
                                            switch (cell.getCellType()) {
     
                                                case Cell.CELL_TYPE_STRING:
                                                    if (row.getRowNum() == 0) { //In the file there is no row number column so I am just inserting one manually
                                                        ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addToHeaderList(cell.getStringCellValue());
                                                    }
                                                    break;
     
                                                default:
                                                    break;
                                            }
     
                                        } 
                                    }
                                    else if (boolIterOnce) // if not traversing the headers
                                    {                      
                                        for (int i = 0; i < maxSize; i++) {
                                            if (i == 0) {
                                                //First column is used for row numbers, convert to string so they are left aligned by default
                                                ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(Integer.toString(row.getRowNum() - 1), row.getRowNum() - 1);
                                            } else {
                                                Cell cell = row.getCell(i - 1, Row.CREATE_NULL_AS_BLANK);
                                                switch (cell.getCellType()) {
                                                    case Cell.CELL_TYPE_NUMERIC: // if the cell is a number
                                                        if (row.getRowNum() == 0) {
                                                        } else {
                                                            if (DateUtil.isCellDateFormatted(cell)) {//speical case when its a date
                                                                ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getDateCellValue(), row.getRowNum() - 1);
                                                            } else {
                                                                ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getNumericCellValue(), row.getRowNum() - 1);
                                                            }
                                                        }
                                                        break;
     
                                                    case Cell.CELL_TYPE_STRING:
                                                        if (row.getRowNum() == 0) {
                                                        } else {
                                                            ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getStringCellValue(), row.getRowNum() - 1);
                                                        }
                                                        break;
                                                    case Cell.CELL_TYPE_BLANK:
                                                        ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell("NO DATA", row.getRowNum() - 1);
                                                        break;
                                                    case Cell.CELL_TYPE_BOOLEAN: // if the cell contains a boolean value                                       
                                                        ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getBooleanCellValue(), row.getRowNum() - 1);
                                                        break;
                                                    case Cell.CELL_TYPE_FORMULA: // if the cell contaions a formula (Shouldn't be called)
                                                        ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).addCell(cell.getCellFormula(), row.getRowNum() - 1);
                                                        break;
                                                    default: 
                                                        System.out.println("unsuported Cell type Row: " + row.getRowNum() + " Column: " + cell.getColumnIndex());
                                                        break;
                                                }
                                            }
                                        }
                                    }
                                    maxSize = ((TableTabInfo) vTabInfo.get(jTabs.getTabCount())).headers.size();
                                    boolIterOnce = true;
                                } 
                            } catch (IOException ex) {
                                ex.printStackTrace();
                                outputTextArea.setText("ERROR: FILE COULD NOT BE OPENED!");
                            }
                        }

    I am using vectors to store each new instance of a TableTabInfo class, which shouldn't be an issue because only a few exist at any given time. I thought it was the data inside that was taking up a lot of space (rData variable perhaps as thats where the raw data is actually stored (in addition to them being loaded into a JTable also in that class. But I set rData to null after loading the file and java.exe remained at 80 MB usage and I understand that setting the object to null essentially deletes it and frees the memory but that did not seem to be the case.

    Any suggestions would be great.

    The problem is occuring when loading .txt files with the same data as well. A 100 MB file takes 1.5 GB in memory using the above code.


  2. #2
    Administrator copeg's Avatar
    Join Date
    Oct 2009
    Location
    US
    Posts
    5,302
    Thanks
    181
    Thanked 824 Times in 767 Posts
    Blog Entries
    5

    Default Re: Loading file into memory is taking up TOO much memory!

    Memory management in java isn't as simple as watching memory usage in the task manager. Memory is managed by a garbage collector - its pretty smart but is not constantly monitoring object references and immediately acting accordingly. Thus setting an object to null (and presuming no more references exist to the object) does not mean it is immediately removed from memory. If you truly wish to inspect memory usage, use a java profiler (for instance VisualVM) to inspect the runtime of your program - from there you can try to nail down what is consuming the most memory, whether you need it, and/or whether you are holding onto references to object you no longer need.

  3. #3
    Junior Member
    Join Date
    Mar 2013
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Re: Loading file into memory is taking up TOO much memory!

    Quote Originally Posted by copeg View Post
    Memory management in java isn't as simple as watching memory usage in the task manager. Memory is managed by a garbage collector - its pretty smart but is not constantly monitoring object references and immediately acting accordingly. Thus setting an object to null (and presuming no more references exist to the object) does not mean it is immediately removed from memory. If you truly wish to inspect memory usage, use a java profiler (for instance VisualVM) to inspect the runtime of your program - from there you can try to nail down what is consuming the most memory, whether you need it, and/or whether you are holding onto references to object you no longer need.
    Thanks. Here are the results from the profiler. They don't seem to add up as the largest classes which are just double and chars show only usage of around 7-8 MB each. In the task manager java.exe is showing as 312 MB used.

    http://i.imgur.com/pZkHoQr.png

  4. #4
    Junior Member
    Join Date
    Mar 2013
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default Re: Loading file into memory is taking up TOO much memory!

    Quote Originally Posted by copeg View Post
    Memory management in java isn't as simple as watching memory usage in the task manager. Memory is managed by a garbage collector - its pretty smart but is not constantly monitoring object references and immediately acting accordingly. Thus setting an object to null (and presuming no more references exist to the object) does not mean it is immediately removed from memory. If you truly wish to inspect memory usage, use a java profiler (for instance VisualVM) to inspect the runtime of your program - from there you can try to nail down what is consuming the most memory, whether you need it, and/or whether you are holding onto references to object you no longer need.
    Thanks, I tried Visual VM and here are the results from the sampler.

    http://i.imgur.com/QhaJSmO.png

    It looks like its showing basic objects - which is hard to tell where exactly they come from.

    These are results from a 27 MB text file. Takes up 650 MB at the start, then drops for some reason.

    See here http://i.imgur.com/cJw2xyU.png

    I really have no clue whats going on, looks like high usage and then a manageable amount. Then it climbs back up.

Similar Threads

  1. Properly releasing memory to avoid memory pileup/crash
    By fickletrick in forum What's Wrong With My Code?
    Replies: 6
    Last Post: July 22nd, 2012, 10:09 AM
  2. [SOLVED] Memory usage increasing in while loop - is it a memory leak
    By mds1256 in forum What's Wrong With My Code?
    Replies: 2
    Last Post: July 18th, 2012, 10:06 AM
  3. In Memory CD Database
    By Laxman2809 in forum What's Wrong With My Code?
    Replies: 4
    Last Post: October 7th, 2011, 11:13 AM
  4. Memory Handling -de allocate memory in Java
    By 19world in forum Java Theory & Questions
    Replies: 4
    Last Post: June 15th, 2010, 04:05 AM
  5. Out of Memory Error
    By wasaki in forum What's Wrong With My Code?
    Replies: 1
    Last Post: May 31st, 2010, 03:37 PM