Forum Moderators: open

Message Too Old, No Replies

Many robots.txt hits but few page hits

         

DanB

6:43 pm on Mar 3, 2005 (gmt 0)

10+ Year Member



When Yahoo Slurp comes to visit it requests my robots.txt file many times but requests other files infrequently. Looking at my logs sometimes that's all slurp requests -- 10am robots.txt , 11am robots.txt, etc..

For March 05 my log shows 47 robot.txt hits and 17 "other" hits by Slurp.

Is it common for Slurp to request robot.txt so many times?

relevant metas:
META NAME="ROBOTS" CONTENT="INDEX, FOLLOW"
META NAME="REVISIT-AFTER" CONTENT="30 DAYS"
META NAME="ROBOTS" CONTENT="NOARCHIVE"

DanB

10:48 pm on Mar 5, 2005 (gmt 0)

10+ Year Member



So now my stats show 122 robots.txt file hits and 32 "other" pages by Slurp.

DanB

3:08 pm on Mar 13, 2005 (gmt 0)

10+ Year Member



Now its at 212 and 109