Welcome to WebmasterWorld Guest from 126.96.36.199
Forum Moderators: goodroi
I have the user-agent set up as * to allow all bots, and disallow
1. What would account for this sudden increase in visits to my robots.txt file?
2.Could there be any connection between the increase in visits to the robots.txt file and the decrease in visits to my other pages?
Most of the visits to robots.txt seem to be coming from Googlebot, Mozilla, Gigabot, ia archiver, seventwentyfour.com and MSNbot.
The number of referrals from Google searches have dropped by about half.
I appreciate any help you can give me.
Checks every link to other domains every day
If there are a lot of links on their domain and their bot is crappy this might result in a lot of robots.txt visits ;).
Also yahoo slurp is very happy to read robots.txt everyday for 10 or 20 times (of our domain).