Thread: Website Source

Hi,
I want to get complete HTML source of a web page ( web page URL given below )
http://shop.safeway.com/superstore/s...sp?page=signin

but the problem I have is that when I get the web page through a java program, I do not get
the complete HTML code that is displayed in the browser. I have tried to get the web page by

three different ways
1. Using HTTpClient class
2. Using URL Connection
3. Using HTML Parser

but the result is same i.e. I do not get the complete HTML source of the page.

Furthermore, when I try to view the page source in any browser, I get exactly the same HTML
source that I get by the Java program. But when I select some part of the page ( like login
part) then I can view the HTML that is not displayed by the Java program. I want to get that HTML in my program


Any help would be highly appreciated

Thanking you

Sounds like you might need to process some Javascript as well then. Maybe you could look into Rhino for this, not exactly sure how all that works though as I've just done what you've tried and just get the HTML data.

// Json

Thanks for your response Json and JavaPF.
Dear JavaPF, I have already tried that example that you sent to me. It does not work
Dear Json, Can you please guide me the way I have to use Rhino. I tried to figure out but could not. I could not find any example related to my case on Rhino website.


Thanking you

I have attached two files.
Source1 File contains page source that I get when I try to view the source in the browser and its the exactly same source that I get through my java program.

Source2 File contains page source when I select/highlight the login form and then try to view the source in the browser, then I get that. I want to get that html source through my java program


Thanks

I've never used Rhino myself so I wouldn't really have a clue to be honest

However looking at your 2 files there, if you have a look at Source1.txt you can see that its creating a page with frames, maybe you should have look at each frame separately.

Topleft: /brands/genericredirecter.asp?Destination=/superstore/leftbar/topLeft.htm
Basket: /superstore/single_addnew.asp
Checkout: /superstore/topbar/defaulttopchkbar.asp?page=shop
Topbar: /superstore/topbar/topbar.asp?pnav=shop
Main: /brands/sslgenericredirecter.asp?destination=%2Fregister%2 Fsignin%2Easp%3Ffrom%3Dsignin%26microsite%3D
Buttons: /superstore/topbar/bottomfooter.asp

OR:

Header: newheader.asp
Main: /brands/sslgenericredirecter.asp?destination=%2Fregister%2 Fsignin%2Easp%3Ffrom%3Dsignin%26microsite%3D
Buttons: /superstore/topbar/bottomfooter.asp


// Json

Dear Json,
Thank you very much. I tried to fetch the source from the frame directly and it worked.
I got the source of page.

I am extremely thankful to your

Regards

No worries, glad to help.

// Json

Oh, I mean to say it did not help me in my problem. Otherwise there was no issue with the link and code posted there.

Thanks