Forum Moderators: open
I know it's good to obey the robots standard, but c'mon Slurp! ;-)
Oh, and it'd be nice to actually get indexed properly (more than 2 copies of my homepage) seeing as I get Slurped at least once a week!
Seems that Slurp has gone mad today
Not sure if your actually making an inquiry or providing a rhetorical insight?
Robot.txt is generally a very small file.
As a rule unless a major bot REPEATEDLY grabbed the file 50 or 100 times would raise any issue of concern with me.
How they SE reacts as to the listing of your pages is an entirely different issue.
Don