Has anyone seen that? I'm seeing about 15 different ip's of Slurp, but all crawlers are only taking the robots.txt file. It has been like that for about 10 days now.
Anyone seen that?
rogerd
10:01 pm on Oct 14, 2004 (gmt 0)
I assume you've checked the robots.txt file to be sure there's not a syntax error that puts the whole site off limits? More than likely, it's an example of weird bot behavior, but I'd be sure to check and doublecheck the file.
