Has slurp gone mental?

[Shri]

I have been seeing similar activity for the last two weeks on my site.

Should I be concerned?

Any ideas?

Shri
---------------------------------------

216.35.116.93 - - [14/Mar/2000:00:16:46 -0800] "GET / HTTP/1.0" 200 25818 "-" "Slurp/si
216.35.116.42 - - [13/Mar/2000:12:34:04 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.42 - - [13/Mar/2000:12:34:05 -0800] "GET / HTTP/1.0" 200 25857 "-" "Slurp/si
216.35.116.92 - - [13/Mar/2000:12:41:47 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.92 - - [13/Mar/2000:12:41:48 -0800] "GET / HTTP/1.0" 200 26115 "-" "Slurp/si
216.35.103.62 - - [13/Mar/2000:13:00:59 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.103.62 - - [13/Mar/2000:13:00:59 -0800] "GET / HTTP/1.0" 200 26113 "-" "Slurp/si
216.35.116.90 - - [13/Mar/2000:13:02:43 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.90 - - [13/Mar/2000:13:02:48 -0800] "GET / HTTP/1.0" 200 26115 "-" "Slurp/si
209.185.143.81 - - [13/Mar/2000:13:56:18 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
209.185.143.81 - - [13/Mar/2000:13:56:19 -0800] "GET / HTTP/1.0" 200 25908 "-" "Slurp/si
216.35.116.41 - - [13/Mar/2000:14:40:55 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.41 - - [13/Mar/2000:14:40:56 -0800] "GET / HTTP/1.0" 200 25939 "-" "Slurp/si
216.35.116.91 - - [13/Mar/2000:14:46:44 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.91 - - [13/Mar/2000:14:46:45 -0800] "GET / HTTP/1.0" 200 26113 "-" "Slurp/si
216.35.116.42 - - [13/Mar/2000:14:58:23 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.42 - - [13/Mar/2000:14:58:24 -0800] "GET / HTTP/1.0" 200 25860 "-" "Slurp/si
216.35.116.92 - - [13/Mar/2000:15:08:53 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.92 - - [13/Mar/2000:15:08:54 -0800] "GET / HTTP/1.0" 200 26121 "-" "Slurp/si
216.35.103.62 - - [13/Mar/2000:15:26:38 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.103.62 - - [13/Mar/2000:15:26:39 -0800] "GET / HTTP/1.0" 200 26121 "-" "Slurp/si
216.35.116.90 - - [13/Mar/2000:15:33:37 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.90 - - [13/Mar/2000:15:33:38 -0800] "GET / HTTP/1.0" 200 25888 "-" "Slurp/si
216.35.103.49 - - [13/Mar/2000:16:41:10 -0800] "GET /ubb/Forum4/HTML/000295.html HTTP/1.0"
216.35.116.41 - - [13/Mar/2000:17:12:16 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.41 - - [13/Mar/2000:17:12:17 -0800] "GET / HTTP/1.0" 200 25701 "-" "Slurp/si
216.35.116.91 - - [13/Mar/2000:17:16:42 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.91 - - [13/Mar/2000:17:16:43 -0800] "GET / HTTP/1.0" 200 25726 "-" "Slurp/si
209.185.143.81 - - [13/Mar/2000:17:29:34 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
209.185.143.81 - - [13/Mar/2000:17:29:35 -0800] "GET / HTTP/1.0" 200 25959 "-" "Slurp/si
216.35.116.74 - - [13/Mar/2000:18:53:02 -0800] "GET /ubb/Forum2/HTML/000107.html HTTP/1.0" 200 29049 "-"
216.35.116.93 - - [13/Mar/2000:21:58:38 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.93 - - [13/Mar/2000:21:58:39 -0800] "GET / HTTP/1.0" 200 25625 "-" "Slurp/si
216.35.103.60 - - [13/Mar/2000:23:56:08 -0800] "GET /ubb/Forum26/HTML/000037.html HTTP/1.0" 200 16378 "-"

[fantomaster]

While it's nothing serious, I *would* suggest generating a robots.txt file to get rid of those annoying 404s.

[Brett_Tabke]

Back in 97(?) when robots.txt first came out, there were some engines that just checked for the presence of the file and aborted it's quest if it found one. Some never even parsed the file. I'd gotten burned a few times by it before I knew what was going on - so, I've not used one since.

Is it really safe to use one now, or are there any brain dead se's left? Ermm, brain dead on robots.txt that is

[fantomaster]

Most big players are honoring the robots.txt convention now. It's only a few minor (mainly European) SEs that
don't, as far as we are aware. Email harvesters are a different story, of course.