Welcome to WebmasterWorld Guest from 54.166.33.25

Forum Moderators: open

Slurp/si bot hitting only robots.txt every day, dropped from index.

Can i hope for reindex?

   
3:10 pm on Jan 14, 2004 (gmt 0)

10+ Year Member



Hi All!
My site has been indexed by Inktomi for free about a half a year ago. Then, about 3 months ago Ink dropped all my pages and i lost all MSN traffic.
Now i have Slurp/si bot visiting my site every day but hitting only robots.txt.
Does it mean that i got some penalty and it's impossible for my site to be indexed for free now?
Thanks in advance.
9:41 am on Jan 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



There could possiblly be an error in your robots.txt, which triggers it to leave your site and comeback another day.

Get your robots.txt file checked at [searchengineworld.com...]

And I know this seams almost impossible, but is your robots.txt or .htaccess banning any of Inktomi's UAs or IPs?
Check them just incase.

Sid

10:04 am on Jan 15, 2004 (gmt 0)

10+ Year Member



Thank you, Sid, for your advice.
I checked my robots.txt - no errors detected.
It's simple:

User-agent: *
Disallow: /admin/
Disallow: /webstat/
Disallow: /test/

My .htaccess is unchanged since site's beginning.
I still can't understand the situation.
?

4:37 pm on Jan 15, 2004 (gmt 0)

10+ Year Member



If your site is dropped because it is banned, it won't help to pay for inclusion. I know from experience on this one. If anybody has had other experience, I would be very interested to hear.
6:57 pm on Jan 15, 2004 (gmt 0)

10+ Year Member



Greenleaf,

Our experience is exactly the same as yours,
and we have been told that our site has picked up
a penalty. The slurp bot stills comes every day,
but only takes robots.txt now. PFI will not help
your pages will be ranked last on any search.
All of our free pages are gone fron the index.

TM

7:06 pm on Jan 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This happened to us too on several domains. All of them had pages listed via PFI until about six months ago. When we didn't renew the PFI, all pages including the non paid ones disappeared within a couple of months and the crawling stopped. Could these penalties have anything to do with dropping the pay program? I'm beginning to think so because we have several other domains that were never included in the PFI and they are crawled and updated on a regular basis even today.
9:05 pm on Jan 15, 2004 (gmt 0)

10+ Year Member



Bad.
Does anybody know what could be the reason for a ban in Ink?
Does it mean that it's a manual ban? Is it for ever?

Tim

8:42 pm on Jan 16, 2004 (gmt 0)

10+ Year Member



If you are banned, you know what the problem was and have fixed it. Please send an email to spamcrusader@inktomiremovethis.com and ask for them to let you back in the index. I would explain what you did wrong and how you have resolved the problem. There is no guarantee that this will get you back in the index but this is the channel we have set up for these sorts of issues.
11:56 pm on Jan 16, 2004 (gmt 0)

10+ Year Member



Thank you very much, Tim, for this address.
The problem is that i don't know what could trigger a ban.
What are the possible reasons? What Ink doesn't like?
Could you be so kind and share your experiences/guesses on that?
Thanks.

Tim

6:08 am on Jan 17, 2004 (gmt 0)

10+ Year Member



There is a content policy listed on Inktomi.com It does not cover every technique as new techniques tend to pop up all the time. Also spam tends to be the extent to which people use seo techniques not just using the technique itself.

Looking through the archives of WebmasterWorld is always a helpful resource too.
Tim

8:28 am on Jan 17, 2004 (gmt 0)

10+ Year Member



Thanks, Tim and all!
12:15 am on Feb 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



sorry to reopen this post,
but looking for answers for my problem Inktomi only spiders robots.txt since 13/1, I found this post, and read the anser from Tim saying: If you are banned, you know what the problem was and have fixed it. Please send an email to spamcrusader@inktomiremovethis.com and ask for them to let you back in the index.

Well I donīt know the problem, but anyway just wrote to that e-mail address, but it isnīt valid, keeps coming back. What can I do please.......

Pd, The new yahoo spider does the same.

Tim

3:35 am on Feb 19, 2004 (gmt 0)

10+ Year Member



Remove the "removethis" from the email address.
8:14 pm on Feb 21, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi Tim,
I wrote, suppose take long to get an answer.
Is there a way to know if you ever been indexed before?
I donīt know if been indexed and dropped, or ever indexed, though before didnīt have stats of site.
This really drives me mad, tried everything, and I donīt have and havnīt ever had any spam on my site, only thing I can imagine is: for years ago I was on an linkfarm, but took away quickly when understood that wasnīt to good, to start with I didnīt know anything so just signed up where I could, and maybe Inktomi ever goes back to check if situation changed.
Just checked the linkfarms and can see sites on there listings that are indexed by Inktomi, so that shouldnīt be the reason either.
Thanks in advance
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month