homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Slurp has a thing for my robots file
slurp has been grabbing my robots file

 6:27 pm on Jan 23, 2004 (gmt 0)


My site dropped out of Inktomi in August last year and hasn't had a single page spidered by slurp since. I lost around 20% of my traffic as a result of this drop!

These last couple of days slurp has been requesting my robots file around 8 - 10 times a day!

Has anyone else had this experience? Could this mean that my site is on its way back in to Ink?




 6:53 pm on Jan 23, 2004 (gmt 0)

Slurp has been pretty thirsty of late, a lot of folks have seen heavy activity.

Has it only requested robots.txt, not anything else? If so, you might want to give it a quick check to make sure you're not inadvertently blocking Slurp.


 7:08 pm on Jan 23, 2004 (gmt 0)

I been trying to figure out way, canīt find reason.
The same happends to me, from 1/1 till 12/1 they didnīt spider the robots.txt at all or any other pages.
And since the 13/1 untill today they are spidering the robots.txt every day, some days only once, other 3-4 and one day 14 days..........

This is my robotfile:
User-agent: *
Disallow: /404.shtml
Disallow: /espanol/404.shtml
Disallow: /svenska/404.shtml
Disallow: /cgi-bin/
Disallow: /scgi-bin/

As well I did save it in linux mode and uploaded it in ascii or something like that, to make sure itīs correctly, but they keep on spidering only the robots.txt. I have another post on it named why does Inktomi spiders me, where you can see I wrote them and they gave me their link to their websearch-guidelines.............


 12:00 am on Jan 24, 2004 (gmt 0)


Try using the robots.txt validator [searchengineworld.com]


 12:09 am on Jan 24, 2004 (gmt 0)

Itīs validated by 3 diferents validator
thanks anyway


 1:05 am on Jan 24, 2004 (gmt 0)

I just read in an spanish site some very interesting thing.

They said, but donīt know why, that when slurp spider the robots.txt file is always like this:

"Mozilla/5.0 (Slurp/si; slurp@inktomi.com; [inktomi.com...]

and when spider rest of site ítīs like this:

"Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]

The diference are in Slurp/si; and Slurp/cat,

just checked, all my spidered robots file are with slurp/si

That means there are two diferent spiders?
and that slurp/cat maybe not started to work properly yet?


 9:35 am on Jan 24, 2004 (gmt 0)

Hi all,

My robots.txt is very basic and valid and it doesn't cause problems for other bots so I don't think its keeping slurp out.

I checked another sites logs and found that slurp is indexing pages with (Slurp/cat) and robots files with (Slurp/si).

Looks like I need to wait for Slurp/si's big brother to visit. Hopefully it's only a matter of time.



 11:47 am on Jan 24, 2004 (gmt 0)

same happens to me:

see also this post in this forum : [webmasterworld.com...]

and this post in another forum : [forums.devshed.com...]

and in another forum :

This has been going on for quite a time now.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved