homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Slurp/si bot hitting only robots.txt every day, dropped from index.
Can i hope for reindex?

 3:10 pm on Jan 14, 2004 (gmt 0)

Hi All!
My site has been indexed by Inktomi for free about a half a year ago. Then, about 3 months ago Ink dropped all my pages and i lost all MSN traffic.
Now i have Slurp/si bot visiting my site every day but hitting only robots.txt.
Does it mean that i got some penalty and it's impossible for my site to be indexed for free now?
Thanks in advance.



 9:41 am on Jan 15, 2004 (gmt 0)

There could possiblly be an error in your robots.txt, which triggers it to leave your site and comeback another day.

Get your robots.txt file checked at [searchengineworld.com...]

And I know this seams almost impossible, but is your robots.txt or .htaccess banning any of Inktomi's UAs or IPs?
Check them just incase.



 10:04 am on Jan 15, 2004 (gmt 0)

Thank you, Sid, for your advice.
I checked my robots.txt - no errors detected.
It's simple:

User-agent: *
Disallow: /admin/
Disallow: /webstat/
Disallow: /test/

My .htaccess is unchanged since site's beginning.
I still can't understand the situation.


 4:37 pm on Jan 15, 2004 (gmt 0)

If your site is dropped because it is banned, it won't help to pay for inclusion. I know from experience on this one. If anybody has had other experience, I would be very interested to hear.


 6:57 pm on Jan 15, 2004 (gmt 0)


Our experience is exactly the same as yours,
and we have been told that our site has picked up
a penalty. The slurp bot stills comes every day,
but only takes robots.txt now. PFI will not help
your pages will be ranked last on any search.
All of our free pages are gone fron the index.



 7:06 pm on Jan 15, 2004 (gmt 0)

This happened to us too on several domains. All of them had pages listed via PFI until about six months ago. When we didn't renew the PFI, all pages including the non paid ones disappeared within a couple of months and the crawling stopped. Could these penalties have anything to do with dropping the pay program? I'm beginning to think so because we have several other domains that were never included in the PFI and they are crawled and updated on a regular basis even today.


 9:05 pm on Jan 15, 2004 (gmt 0)

Does anybody know what could be the reason for a ban in Ink?
Does it mean that it's a manual ban? Is it for ever?


 8:42 pm on Jan 16, 2004 (gmt 0)

If you are banned, you know what the problem was and have fixed it. Please send an email to spamcrusader@inktomiremovethis.com and ask for them to let you back in the index. I would explain what you did wrong and how you have resolved the problem. There is no guarantee that this will get you back in the index but this is the channel we have set up for these sorts of issues.


 11:56 pm on Jan 16, 2004 (gmt 0)

Thank you very much, Tim, for this address.
The problem is that i don't know what could trigger a ban.
What are the possible reasons? What Ink doesn't like?
Could you be so kind and share your experiences/guesses on that?


 6:08 am on Jan 17, 2004 (gmt 0)

There is a content policy listed on Inktomi.com It does not cover every technique as new techniques tend to pop up all the time. Also spam tends to be the extent to which people use seo techniques not just using the technique itself.

Looking through the archives of WebmasterWorld is always a helpful resource too.


 8:28 am on Jan 17, 2004 (gmt 0)

Thanks, Tim and all!


 12:15 am on Feb 19, 2004 (gmt 0)

sorry to reopen this post,
but looking for answers for my problem Inktomi only spiders robots.txt since 13/1, I found this post, and read the anser from Tim saying: If you are banned, you know what the problem was and have fixed it. Please send an email to spamcrusader@inktomiremovethis.com and ask for them to let you back in the index.

Well I donīt know the problem, but anyway just wrote to that e-mail address, but it isnīt valid, keeps coming back. What can I do please.......

Pd, The new yahoo spider does the same.


 3:35 am on Feb 19, 2004 (gmt 0)

Remove the "removethis" from the email address.


 8:14 pm on Feb 21, 2004 (gmt 0)

Hi Tim,
I wrote, suppose take long to get an answer.
Is there a way to know if you ever been indexed before?
I donīt know if been indexed and dropped, or ever indexed, though before didnīt have stats of site.
This really drives me mad, tried everything, and I donīt have and havnīt ever had any spam on my site, only thing I can imagine is: for years ago I was on an linkfarm, but took away quickly when understood that wasnīt to good, to start with I didnīt know anything so just signed up where I could, and maybe Inktomi ever goes back to check if situation changed.
Just checked the linkfarms and can see sites on there listings that are indexed by Inktomi, so that shouldnīt be the reason either.
Thanks in advance

Global Options:
 top home search open messages active posts  

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved