Forum Moderators: goodroi
Just wondering how long before the three major SEs index and start using the data from a robots.txt file.
As a guide, Google usually takes about 3-7 days between a page going live on my site and including it in their index. Should I assume the same for their acknowledgment of my robots file or longer?
This is why many people have a terrible time getting their sites indexed. They have errors or mistakes in their robots.txt and they continue to instruct bots to go away or ignore their sites.
I was just wondering if anyone had a guesstimate about how long after accessing a robots.txt file, it takes the SEs to actually implement the information they have gleaned from the robots file.
The bots that behave take note & obey your robots.txt the first time they touch your site. They will check your robots.txt each time they return to make sure they are being polite and to see if you have changed your behavior requests.
The evidence from my own log files suggests that well behaved spiders don't check robots.txt quite so frequently.
Matt
