Welcome to WebmasterWorld Guest from 54.167.40.25

Forum Moderators: open

Message Too Old, No Replies

Has slurp gone mental?

     
8:39 am on Mar 14, 2000 (gmt 0)

Senior Member from HK 

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 14, 2002
posts:2283
votes: 10


I have been seeing similar activity for the last two weeks on my site.

Should I be concerned?

Any ideas?

Shri
---------------------------------------

216.35.116.93 - - [14/Mar/2000:00:16:46 -0800] "GET / HTTP/1.0" 200 25818 "-" "Slurp/si
216.35.116.42 - - [13/Mar/2000:12:34:04 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.42 - - [13/Mar/2000:12:34:05 -0800] "GET / HTTP/1.0" 200 25857 "-" "Slurp/si
216.35.116.92 - - [13/Mar/2000:12:41:47 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.92 - - [13/Mar/2000:12:41:48 -0800] "GET / HTTP/1.0" 200 26115 "-" "Slurp/si
216.35.103.62 - - [13/Mar/2000:13:00:59 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.103.62 - - [13/Mar/2000:13:00:59 -0800] "GET / HTTP/1.0" 200 26113 "-" "Slurp/si
216.35.116.90 - - [13/Mar/2000:13:02:43 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.90 - - [13/Mar/2000:13:02:48 -0800] "GET / HTTP/1.0" 200 26115 "-" "Slurp/si
209.185.143.81 - - [13/Mar/2000:13:56:18 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
209.185.143.81 - - [13/Mar/2000:13:56:19 -0800] "GET / HTTP/1.0" 200 25908 "-" "Slurp/si
216.35.116.41 - - [13/Mar/2000:14:40:55 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.41 - - [13/Mar/2000:14:40:56 -0800] "GET / HTTP/1.0" 200 25939 "-" "Slurp/si
216.35.116.91 - - [13/Mar/2000:14:46:44 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.91 - - [13/Mar/2000:14:46:45 -0800] "GET / HTTP/1.0" 200 26113 "-" "Slurp/si
216.35.116.42 - - [13/Mar/2000:14:58:23 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.42 - - [13/Mar/2000:14:58:24 -0800] "GET / HTTP/1.0" 200 25860 "-" "Slurp/si
216.35.116.92 - - [13/Mar/2000:15:08:53 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.92 - - [13/Mar/2000:15:08:54 -0800] "GET / HTTP/1.0" 200 26121 "-" "Slurp/si
216.35.103.62 - - [13/Mar/2000:15:26:38 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.103.62 - - [13/Mar/2000:15:26:39 -0800] "GET / HTTP/1.0" 200 26121 "-" "Slurp/si
216.35.116.90 - - [13/Mar/2000:15:33:37 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.90 - - [13/Mar/2000:15:33:38 -0800] "GET / HTTP/1.0" 200 25888 "-" "Slurp/si
216.35.103.49 - - [13/Mar/2000:16:41:10 -0800] "GET /ubb/Forum4/HTML/000295.html HTTP/1.0"
216.35.116.41 - - [13/Mar/2000:17:12:16 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.41 - - [13/Mar/2000:17:12:17 -0800] "GET / HTTP/1.0" 200 25701 "-" "Slurp/si
216.35.116.91 - - [13/Mar/2000:17:16:42 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.91 - - [13/Mar/2000:17:16:43 -0800] "GET / HTTP/1.0" 200 25726 "-" "Slurp/si
209.185.143.81 - - [13/Mar/2000:17:29:34 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
209.185.143.81 - - [13/Mar/2000:17:29:35 -0800] "GET / HTTP/1.0" 200 25959 "-" "Slurp/si
216.35.116.74 - - [13/Mar/2000:18:53:02 -0800] "GET /ubb/Forum2/HTML/000107.html HTTP/1.0" 200 29049 "-"
216.35.116.93 - - [13/Mar/2000:21:58:38 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.93 - - [13/Mar/2000:21:58:39 -0800] "GET / HTTP/1.0" 200 25625 "-" "Slurp/si
216.35.103.60 - - [13/Mar/2000:23:56:08 -0800] "GET /ubb/Forum26/HTML/000037.html HTTP/1.0" 200 16378 "-"

5:12 am on Mar 15, 2000 (gmt 0)

Full Member

10+ Year Member

joined:Sept 23, 1999
posts:201
votes: 0


While it's nothing serious, I *would* suggest generating a robots.txt file to get rid of those annoying 404s.
11:32 am on Mar 30, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


Back in 97(?) when robots.txt first came out, there were some engines that just checked for the presence of the file and aborted it's quest if it found one. Some never even parsed the file. I'd gotten burned a few times by it before I knew what was going on - so, I've not used one since.

Is it really safe to use one now, or are there any brain dead se's left? Ermm, brain dead on robots.txt that is

5:17 pm on Mar 30, 2000 (gmt 0)

Full Member

10+ Year Member

joined:Sept 23, 1999
posts:201
votes: 0


Most big players are honoring the robots.txt convention now. It's only a few minor (mainly European) SEs that
don't, as far as we are aware. Email harvesters are a different story, of course.