homepage Welcome to WebmasterWorld Guest from 54.211.190.232
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderators: Brett Tabke

Paid Inclusion Engines and Topics Forum

  posting off  
Has slurp gone mental?
Shri




msg:29557
 8:39 am on Mar 14, 2000 (gmt 0)

I have been seeing similar activity for the last two weeks on my site.

Should I be concerned?

Any ideas?

Shri
---------------------------------------

216.35.116.93 - - [14/Mar/2000:00:16:46 -0800] "GET / HTTP/1.0" 200 25818 "-" "Slurp/si
216.35.116.42 - - [13/Mar/2000:12:34:04 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.42 - - [13/Mar/2000:12:34:05 -0800] "GET / HTTP/1.0" 200 25857 "-" "Slurp/si
216.35.116.92 - - [13/Mar/2000:12:41:47 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.92 - - [13/Mar/2000:12:41:48 -0800] "GET / HTTP/1.0" 200 26115 "-" "Slurp/si
216.35.103.62 - - [13/Mar/2000:13:00:59 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.103.62 - - [13/Mar/2000:13:00:59 -0800] "GET / HTTP/1.0" 200 26113 "-" "Slurp/si
216.35.116.90 - - [13/Mar/2000:13:02:43 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.90 - - [13/Mar/2000:13:02:48 -0800] "GET / HTTP/1.0" 200 26115 "-" "Slurp/si
209.185.143.81 - - [13/Mar/2000:13:56:18 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
209.185.143.81 - - [13/Mar/2000:13:56:19 -0800] "GET / HTTP/1.0" 200 25908 "-" "Slurp/si
216.35.116.41 - - [13/Mar/2000:14:40:55 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.41 - - [13/Mar/2000:14:40:56 -0800] "GET / HTTP/1.0" 200 25939 "-" "Slurp/si
216.35.116.91 - - [13/Mar/2000:14:46:44 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.91 - - [13/Mar/2000:14:46:45 -0800] "GET / HTTP/1.0" 200 26113 "-" "Slurp/si
216.35.116.42 - - [13/Mar/2000:14:58:23 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.42 - - [13/Mar/2000:14:58:24 -0800] "GET / HTTP/1.0" 200 25860 "-" "Slurp/si
216.35.116.92 - - [13/Mar/2000:15:08:53 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.92 - - [13/Mar/2000:15:08:54 -0800] "GET / HTTP/1.0" 200 26121 "-" "Slurp/si
216.35.103.62 - - [13/Mar/2000:15:26:38 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.103.62 - - [13/Mar/2000:15:26:39 -0800] "GET / HTTP/1.0" 200 26121 "-" "Slurp/si
216.35.116.90 - - [13/Mar/2000:15:33:37 -0800] "GET /robots.txt HTTP/1.0" 404 - "-
216.35.116.90 - - [13/Mar/2000:15:33:38 -0800] "GET / HTTP/1.0" 200 25888 "-" "Slurp/si
216.35.103.49 - - [13/Mar/2000:16:41:10 -0800] "GET /ubb/Forum4/HTML/000295.html HTTP/1.0"
216.35.116.41 - - [13/Mar/2000:17:12:16 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.41 - - [13/Mar/2000:17:12:17 -0800] "GET / HTTP/1.0" 200 25701 "-" "Slurp/si
216.35.116.91 - - [13/Mar/2000:17:16:42 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.91 - - [13/Mar/2000:17:16:43 -0800] "GET / HTTP/1.0" 200 25726 "-" "Slurp/si
209.185.143.81 - - [13/Mar/2000:17:29:34 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
209.185.143.81 - - [13/Mar/2000:17:29:35 -0800] "GET / HTTP/1.0" 200 25959 "-" "Slurp/si
216.35.116.74 - - [13/Mar/2000:18:53:02 -0800] "GET /ubb/Forum2/HTML/000107.html HTTP/1.0" 200 29049 "-"
216.35.116.93 - - [13/Mar/2000:21:58:38 -0800] "GET /robots.txt HTTP/1.0" 404 - "-"
216.35.116.93 - - [13/Mar/2000:21:58:39 -0800] "GET / HTTP/1.0" 200 25625 "-" "Slurp/si
216.35.103.60 - - [13/Mar/2000:23:56:08 -0800] "GET /ubb/Forum26/HTML/000037.html HTTP/1.0" 200 16378 "-"

 

fantomaster




msg:29558
 5:12 am on Mar 15, 2000 (gmt 0)

While it's nothing serious, I *would* suggest generating a robots.txt file to get rid of those annoying 404s.

Brett_Tabke




msg:29559
 11:32 am on Mar 30, 2000 (gmt 0)

Back in 97(?) when robots.txt first came out, there were some engines that just checked for the presence of the file and aborted it's quest if it found one. Some never even parsed the file. I'd gotten burned a few times by it before I knew what was going on - so, I've not used one since.

Is it really safe to use one now, or are there any brain dead se's left? Ermm, brain dead on robots.txt that is

fantomaster




msg:29560
 5:17 pm on Mar 30, 2000 (gmt 0)

Most big players are honoring the robots.txt convention now. It's only a few minor (mainly European) SEs that
don't, as far as we are aware. Email harvesters are a different story, of course.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved