spidering speed - (deprecated) Altavista, Alltheweb.com forum at WebmasterWorld - WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

spidering speed

How fast is fast...

pardo

12:48 pm on Dec 29, 2002 (gmt 0)

10+ Year Member

Seems that I have been visiting by the Fast Webcrawler, it only picked up my robots.txt file.

When is it likely that it will crawl through the html-pages?

Brett_Tabke

2:06 pm on Dec 31, 2002 (gmt 0)

WebmasterWorld Administrator

10+ Year Member

Top Contributors Of The Month

It should happen within 7 days to two weeks. If it is a new site, it may be up to one month before it is picked up entirely.