Java XML parsing for HUGE size file and no root tags

[peekuabc]

Hi All,

I am working on situation where i have to write code to parse XML files with following :

1) Size of file will be Huge in future (So the code should be smart enough to deal with it). It may maximum reach upto 8GB.
2) The XML is not proper as it has no fix root tag rather have multiple XML in a single XML. (So we may need to add root tag and delete repated <?xml version and <!DOCTYPE ? tags) or split into multiple xml before parsing.
3) There is no XSD provide by the client rather DTD is given.

Do anyone has anything to share for such problem statement. I am thinking of multiple approach to deal with this situation like using shell script, SAX or sTax etc.

Just came to this forum, to see if we have similar problem faced by someone on future and i can be lucky

[copeg]

This thread has been cross posted here:

http://www.java-forums.org/xml/79540-java-xml-parsing-huge-size-file-no-root-tags.html

Although cross posting is allowed, for everyone's benefit, please read:

Java Programming Forums Cross Posting Rules

The Problems With Cross Posting

[aussiemcgr]

Out of curiosity: why use xml if you are ignoring some of the basic standards (like root nodes) and enormous file sizes?
I'm asking because I suspect you may be attempting to do something in xml which may be far more suited to do in some other fashion.

[peekuabc]

Out of curiosity: why use xml if you are ignoring some of the basic standards (like root nodes) and enormous file sizes?
I'm asking because I suspect you may be attempting to do something in xml which may be far more suited to do in some other fashion.

Hi aussiemcgr,

We are getting this xml from the some other application with which we are going to integrate so this is what the requirement is and we have to use the same XML.

We have to parse this XML and finally inster the date from it into out database after validation.

Please let me know if you have some views.

thanks...

[aussiemcgr]

So is the file itself an xml, or is a text file which contains a handful of xml structures? I ask, because if it is the latter, you will probably not be able to read it with any native xml reader.

Also, if it is the latter, my immediate suggestion (which may not be the "best" way of doing it) would be to create a handful of temp files and put each xml structure in their own temp file, which you can then read independently as xmls. I suggest this because the large file that you get from the other program will have to be parsed with a stream buffer. As far as I know, there is no other way since you cannot store 8gb in memory (unless you have a really powerful computer and you are able to super-jack the amount of available memory java is allowed to access somehow). So, if you break up each structure into their own temp xml files, you may be able to load each xml file into memory independently while you parse them; or, at the very least, be able to use one of the java xml parsers. Thoughts?

[peekuabc]

Hi,

It's a XML file only. we r thinking of 2 approaches:

1) One thing to manipulate XML by adding root tags and deleted some repeated stuff and parse it using SAX or Stax as they don't load everything in memory ...
2) As you said break it up, put in temp folder and than process it one by one. (looks to be right choice if option 1 don't give good response in terms of performance)

thanks