Welcome to WebmasterWorld Guest from

Forum Moderators: ergophobe

Message Too Old, No Replies

Indexing your content

web spiders, sitemaps, index



3:31 pm on Aug 2, 2006 (gmt 0)

10+ Year Member

Does anyone know of:

1. A reliable spider that will INDEX ONLY (don't need full web download) a site over HTTP:

2. A tool with a method to capture the results of site index search (such as site:www.mydomain.com in Google)into a text file.

I've tried close to about 10 tools over the past 3 days and all of them are less than reliable. This is my last effort before I look to grab the code and write my own. Any info would be appreciated, thanks


Featured Threads

Hot Threads This Week

Hot Threads This Month