Creating a category specific crawler

crawler, search engine

[rodriguez1804]

Hey guys,

I am trying to create a crawler that goes through a list of websites for a certain category, say food. I also don't want to reinvent the wheel. Is this possible? Is there anything out there similar to this already? If not, what would be the best way to go about it? Thanks for the help.

Also, I ve been wondering, how do you get a list of all the websites out there available to crawl?

[enigma1]

if you're looking for code, there are lots of open source packages where the principles of searching, indexing etc are implemented, you could check in sourceforge.net for instance.