How to Make Website Crawler

Look for a link to my website

[wfernley]

Hello,

I was curious if it is easy to create a website crawler that will crawl a website I specify and find a link to my website. Is there a tutorial out there that can help me build a crawler?

Thanks :)

Wes

[Aleister]

I have not seen such a tutorial, but then again, I have not ever looked. You might be able to use a program like Xenu though. It is mainly used to search an entire site for broken links, but it has other uses. It generates a complete list of pages on the site, as well as a list of out-bound links. I belive you can save the results to a file as well.

You could get a list of all the out-bound links the site contains (which could be very large for a big site), and search for your link in the list.

It has other options, so it may have a way to make the process even easier, although I have never tried it. It does make finding broken links easy though in any case, which is usefull :)

[edited by: jatar_k at 5:14 pm (utc) on Aug. 24, 2005]
[edit reason] no urls thanks [/edit]