Welcome to WebmasterWorld Guest from

Forum Moderators: ergophobe

Message Too Old, No Replies

Indexing your content

web spiders, sitemaps, index

3:31 pm on Aug 2, 2006 (gmt 0)

New User

10+ Year Member

joined:June 16, 2005
votes: 0

Does anyone know of:

1. A reliable spider that will INDEX ONLY (don't need full web download) a site over HTTP:

2. A tool with a method to capture the results of site index search (such as site:www.mydomain.com in Google)into a text file.

I've tried close to about 10 tools over the past 3 days and all of them are less than reliable. This is my last effort before I look to grab the code and write my own. Any info would be appreciated, thanks


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members