homepage Welcome to WebmasterWorld Guest from 54.205.189.156
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Sudden Increase in robots.txt Hits
Rob_Moore




msg:1526295
 11:56 am on Dec 13, 2004 (gmt 0)

The past month or so the number of visits to my robot.txt file has increased dramatically. Before, Id get a few visits a week, but now Im suddenly getting about 500 visits a week to the file (it is in fact my most-visited page right now). At the same time, visits to my other pages have dropped by about a third (to about 3,500 visits a week).

I have the user-agent set up as * to allow all bots, and disallow

/cgi-bin/
/stats/
/cgi-image/
/webmail/

1. What would account for this sudden increase in visits to my robots.txt file?

2.Could there be any connection between the increase in visits to the robots.txt file and the decrease in visits to my other pages?

Most of the visits to robots.txt seem to be coming from Googlebot, Mozilla, Gigabot, ia archiver, seventwentyfour.com and MSNbot.

The number of referrals from Google searches have dropped by about half.

I appreciate any help you can give me.

Rob

 

teaperson




msg:1526296
 6:06 pm on Dec 20, 2004 (gmt 0)

Did this coincide with any changes to the robots.txt file?

Could it be that another site added links to your site, which meant that a bunch of new crawlers have discovered your site?

mars9820




msg:1526297
 10:04 pm on Dec 26, 2004 (gmt 0)

maybe one of your linkpartners signed up for the free trial at seventwentyfour.com because they have the following feature:

Checks every link to other domains every day

If there are a lot of links on their domain and their bot is crappy this might result in a lot of robots.txt visits ;).

Also yahoo slurp is very happy to read robots.txt everyday for 10 or 20 times (of our domain).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved