Msg#: 3031621 posted 3:31 pm on Aug 2, 2006 (gmt 0)
Does anyone know of:
1. A reliable spider that will INDEX ONLY (don't need full web download) a site over HTTP:
2. A tool with a method to capture the results of site index search (such as site:www.mydomain.com in Google)into a text file.
I've tried close to about 10 tools over the past 3 days and all of them are less than reliable. This is my last effort before I look to grab the code and write my own. Any info would be appreciated, thanks