Get "Object Moved" error when using url.openConnection and .connect
I have been using Matlab Java Virtual Machine (Java 1.6.0_27-b07 with Sun Microsystems Inc. Java HotSpot(TM) Client VM mixed mode) for the past 2 years to login to an https site to look for high school teaching substitute job postings, and accept jobs that I would want. Everything was working great until a couple of months ago. I started getting an error when I ran the program. I looked deeper and saw that I was still able to securely log into the site, but when I went to the job search page, it read:
<body><h1>Object Moved</h1>This object may be found <a HREF="http://www.aesoponline.com/login.asp">here</a>.</body>"
The problem is that http://www.aesoponline.com/login.asp is not the page I want to go to. When I use any browser to go to the search page, I get the correct page and there is no "Object Moved" error.
I have read this may happen if the server thinks you are not a normal browser (an automated computer program trying to access the site). So I have tried adding:
"urlConnection.setRequestProperty('User-agent','Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)');"
to make the server think it is a browser. But this did not help.
I also tried:
to see if it would redirect, but this also did not work.
The strange part is that this all of a sudden started happening a couple months ago, when it worked fine for 2 years before that.
I should mention I am using Matlab because I need to use the image proccessing toolbox because a captcha is used when I accept jobs.
I would appreciate any help I can get. Thanks in advance!
Here is the code I am using after logging into the site and getting the correct cookies ("my_cookie"). "search_page_source_code" is where I read "Object Moved...":
url = java.net.URL('https://www.aesoponline.com/subweb/sub_searchforjobs.asp?x=x');
urlConnection = url.openConnection();
%% Add cookie info
%% Get source code of search page
inputStream = urlConnection.getInputStream;
byteArrayOutputStream = java.io.ByteArrayOutputStream;
% This StreamCopier is unsupported and may change at any time.
isc = InterruptibleStreamCopier.getInterruptibleStreamCo pier;
search_page_source_code = native2unicode(typecast(byteArrayOutputStream.toBy teArray','uint8'),'UTF-8');
Re: Get "Object Moved" error when using url.openConnection and .connect
Your only hope may be to write some kind of proxy to look at all the web traffic between browser and server (some browsers have good HTTP header reporting), or use something like Wireshark to examine the HTTP traffic and see where your bot differs from your browser. Are you adequately handling header fields for cookies and referers?