Welcome to WebmasterWorld Guest from 54.147.20.131

Forum Moderators: open

Message Too Old, No Replies

Has slurp gone mental?

     

Shri

8:39 am on Mar 14, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have been seeing similar activity for the last two weeks on my site.

Should I be concerned?

Any ideas?

Shri
---------------------------------------

216.35.116.93 - - [14/Mar/2000:00:16:46 -0800] "GET / HTTP/1.0" 200 25818 "-" "Slurp/si
216.35.116.42 - - [13/Mar/2000:12:34:04 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.42 - - [13/Mar/2000:12:34:05 -0800] "GET / HTTP/1.0" 200 25857 "-" "Slurp/si
216.35.116.92 - - [13/Mar/2000:12:41:47 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.92 - - [13/Mar/2000:12:41:48 -0800] "GET / HTTP/1.0" 200 26115 "-" "Slurp/si
216.35.103.62 - - [13/Mar/2000:13:00:59 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.103.62 - - [13/Mar/2000:13:00:59 -0800] "GET / HTTP/1.0" 200 26113 "-" "Slurp/si
216.35.116.90 - - [13/Mar/2000:13:02:43 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.90 - - [13/Mar/2000:13:02:48 -0800] "GET / HTTP/1.0" 200 26115 "-" "Slurp/si
209.185.143.81 - - [13/Mar/2000:13:56:18 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
209.185.143.81 - - [13/Mar/2000:13:56:19 -0800] "GET / HTTP/1.0" 200 25908 "-" "Slurp/si
216.35.116.41 - - [13/Mar/2000:14:40:55 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.41 - - [13/Mar/2000:14:40:56 -0800] "GET / HTTP/1.0" 200 25939 "-" "Slurp/si
216.35.116.91 - - [13/Mar/2000:14:46:44 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.91 - - [13/Mar/2000:14:46:45 -0800] "GET / HTTP/1.0" 200 26113 "-" "Slurp/si
216.35.116.42 - - [13/Mar/2000:14:58:23 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.42 - - [13/Mar/2000:14:58:24 -0800] "GET / HTTP/1.0" 200 25860 "-" "Slurp/si
216.35.116.92 - - [13/Mar/2000:15:08:53 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.92 - - [13/Mar/2000:15:08:54 -0800] "GET / HTTP/1.0" 200 26121 "-" "Slurp/si
216.35.103.62 - - [13/Mar/2000:15:26:38 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.103.62 - - [13/Mar/2000:15:26:39 -0800] "GET / HTTP/1.0" 200 26121 "-" "Slurp/si
216.35.116.90 - - [13/Mar/2000:15:33:37 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.90 - - [13/Mar/2000:15:33:38 -0800] "GET / HTTP/1.0" 200 25888 "-" "Slurp/si
216.35.103.49 - - [13/Mar/2000:16:41:10 -0800] "GET /ubb/Forum4/HTML/000295.html HTTP/1.0"
216.35.116.41 - - [13/Mar/2000:17:12:16 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.41 - - [13/Mar/2000:17:12:17 -0800] "GET / HTTP/1.0" 200 25701 "-" "Slurp/si
216.35.116.91 - - [13/Mar/2000:17:16:42 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.91 - - [13/Mar/2000:17:16:43 -0800] "GET / HTTP/1.0" 200 25726 "-" "Slurp/si
209.185.143.81 - - [13/Mar/2000:17:29:34 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
209.185.143.81 - - [13/Mar/2000:17:29:35 -0800] "GET / HTTP/1.0" 200 25959 "-" "Slurp/si
216.35.116.74 - - [13/Mar/2000:18:53:02 -0800] "GET /ubb/Forum2/HTML/000107.html HTTP/1.0" 200 29049 "-"
216.35.116.93 - - [13/Mar/2000:21:58:38 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.93 - - [13/Mar/2000:21:58:39 -0800] "GET / HTTP/1.0" 200 25625 "-" "Slurp/si
216.35.103.60 - - [13/Mar/2000:23:56:08 -0800] "GET /ubb/Forum26/HTML/000037.html HTTP/1.0" 200 16378 "-"

fantomaster

5:12 am on Mar 15, 2000 (gmt 0)

10+ Year Member



While it's nothing serious, I *would* suggest generating a robots.txt file to get rid of those annoying 404s.

Brett_Tabke

11:32 am on Mar 30, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Back in 97(?) when robots.txt first came out, there were some engines that just checked for the presence of the file and aborted it's quest if it found one. Some never even parsed the file. I'd gotten burned a few times by it before I knew what was going on - so, I've not used one since.

Is it really safe to use one now, or are there any brain dead se's left? Ermm, brain dead on robots.txt that is

fantomaster

5:17 pm on Mar 30, 2000 (gmt 0)

10+ Year Member



Most big players are honoring the robots.txt convention now. It's only a few minor (mainly European) SEs that
don't, as far as we are aware. Email harvesters are a different story, of course.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month