java crawler

[anuu]

salamzz my final year project is java crawler nd em confused related to starting plzz help me..which one is the best algo for the crawler

[angstrem]

Consider this course. They'll teach you the basics of programming on Python on a web crawler example.

[anuu]

bt python is different language and i have to make web crawler in java bt cn't understand frm where i start my work

[angstrem]

Doesn't matter. They will show you an algorithm and explain it, so that you know where to start. Crawler is a trivial project, who can be implemented just in one simple function, and if you know the material you were supposed to learn this year, you'll definitely be able to do it.

[anuu]

okhzz thnkss sir can you send m the useful link frm where i gt the usefull inf..r which algo is best for the crawler

[angstrem]

this

[anuu]

one of my uni sir said use specificc website frm where u want to access data bt other teachers said k thats nt possible.r i js want relevent data nt other stuff frm the otherweb sites..

[angstrem]

The relevancy can be achieved by using filtering function during graph traversal.

[albsmith]

Hi,
You can write a crawler using java with the help of existing API’s. You can use the below algorithm to start you java crawler final year project. while(list of unvisited URLs is not empty) {
take URL from list
fetch content
record whatever it is you want to about the content
if content is HTML {
parse out URLs from links
foreach URL {
if it matches your rules
and it's not already in either the visited or unvisited list
add it to the unvisited list
}
}
}

[GregBrannon]

Please, please, please, don't resurrect dead threads. Check the date the thread was started AND the date of the last post. If they're not within the past 2 or 3 weeks, the thread isn't relevant anymore.

Closing this one.