Forum Moderators: open

Message Too Old, No Replies

Slurp is a great name...

It's gone mad and is slurping my robots.txt

         

badass101

12:19 am on Jun 8, 2005 (gmt 0)

10+ Year Member



Seems that Slurp has gone mad today and is hitting my robots.txt 10-15 times in a 20 second period from different IP's.
It's then proceeded to take a few pages throughout the day, and a whole load more hits on robots.txt

I know it's good to obey the robots standard, but c'mon Slurp! ;-)
Oh, and it'd be nice to actually get indexed properly (more than 2 copies of my homepage) seeing as I get Slurped at least once a week!

wilderness

4:03 pm on Jun 8, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Seems that Slurp has gone mad today

Not sure if your actually making an inquiry or providing a rhetorical insight?

Robot.txt is generally a very small file.
As a rule unless a major bot REPEATEDLY grabbed the file 50 or 100 times would raise any issue of concern with me.

How they SE reacts as to the listing of your pages is an entirely different issue.

Don