Welcome to WebmasterWorld Guest from 54.166.33.25

Forum Moderators: ergophobe

Indexing your content

web spiders, sitemaps, index

   
3:31 pm on Aug 2, 2006 (gmt 0)

5+ Year Member



Does anyone know of:

1. A reliable spider that will INDEX ONLY (don't need full web download) a site over HTTP:

2. A tool with a method to capture the results of site index search (such as site:www.mydomain.com in Google)into a text file.

I've tried close to about 10 tools over the past 3 days and all of them are less than reliable. This is my last effort before I look to grab the code and write my own. Any info would be appreciated, thanks

 

Featured Threads

Hot Threads This Week

Hot Threads This Month