Welcome to WebmasterWorld Guest from 54.242.231.210

Forum Moderators: open

Message Too Old, No Replies

Slurp/si bot hitting only robots.txt every day, dropped from index.

Can i hope for reindex?

     
3:10 pm on Jan 14, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 10, 2003
posts:161
votes: 0


Hi All!
My site has been indexed by Inktomi for free about a half a year ago. Then, about 3 months ago Ink dropped all my pages and i lost all MSN traffic.
Now i have Slurp/si bot visiting my site every day but hitting only robots.txt.
Does it mean that i got some penalty and it's impossible for my site to be indexed for free now?
Thanks in advance.
9:41 am on Jan 15, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 11, 2003
posts:955
votes: 0


There could possiblly be an error in your robots.txt, which triggers it to leave your site and comeback another day.

Get your robots.txt file checked at [searchengineworld.com...]

And I know this seams almost impossible, but is your robots.txt or .htaccess banning any of Inktomi's UAs or IPs?
Check them just incase.

Sid

10:04 am on Jan 15, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 10, 2003
posts:161
votes: 0


Thank you, Sid, for your advice.
I checked my robots.txt - no errors detected.
It's simple:

User-agent: *
Disallow: /admin/
Disallow: /webstat/
Disallow: /test/

My .htaccess is unchanged since site's beginning.
I still can't understand the situation.
?

4:37 pm on Jan 15, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 25, 2000
posts:174
votes: 0


If your site is dropped because it is banned, it won't help to pay for inclusion. I know from experience on this one. If anybody has had other experience, I would be very interested to hear.
6:57 pm on Jan 15, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 1, 2003
posts:62
votes: 0


Greenleaf,

Our experience is exactly the same as yours,
and we have been told that our site has picked up
a penalty. The slurp bot stills comes every day,
but only takes robots.txt now. PFI will not help
your pages will be ranked last on any search.
All of our free pages are gone fron the index.

TM

7:06 pm on Jan 15, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:742
votes: 0


This happened to us too on several domains. All of them had pages listed via PFI until about six months ago. When we didn't renew the PFI, all pages including the non paid ones disappeared within a couple of months and the crawling stopped. Could these penalties have anything to do with dropping the pay program? I'm beginning to think so because we have several other domains that were never included in the PFI and they are crawled and updated on a regular basis even today.
9:05 pm on Jan 15, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 10, 2003
posts:161
votes: 0


Bad.
Does anybody know what could be the reason for a ban in Ink?
Does it mean that it's a manual ban? Is it for ever?

Tim

8:42 pm on Jan 16, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 27, 2002
posts:351
votes: 0


If you are banned, you know what the problem was and have fixed it. Please send an email to spamcrusader@inktomiremovethis.com and ask for them to let you back in the index. I would explain what you did wrong and how you have resolved the problem. There is no guarantee that this will get you back in the index but this is the channel we have set up for these sorts of issues.
11:56 pm on Jan 16, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 10, 2003
posts:161
votes: 0


Thank you very much, Tim, for this address.
The problem is that i don't know what could trigger a ban.
What are the possible reasons? What Ink doesn't like?
Could you be so kind and share your experiences/guesses on that?
Thanks.

Tim

6:08 am on Jan 17, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 27, 2002
posts:351
votes: 0


There is a content policy listed on Inktomi.com It does not cover every technique as new techniques tend to pop up all the time. Also spam tends to be the extent to which people use seo techniques not just using the technique itself.

Looking through the archives of WebmasterWorld is always a helpful resource too.
Tim

8:28 am on Jan 17, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 10, 2003
posts:161
votes: 0


Thanks, Tim and all!
12:15 am on Feb 19, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 27, 2003
posts:1308
votes: 0


sorry to reopen this post,
but looking for answers for my problem Inktomi only spiders robots.txt since 13/1, I found this post, and read the anser from Tim saying: If you are banned, you know what the problem was and have fixed it. Please send an email to spamcrusader@inktomiremovethis.com and ask for them to let you back in the index.

Well I donīt know the problem, but anyway just wrote to that e-mail address, but it isnīt valid, keeps coming back. What can I do please.......

Pd, The new yahoo spider does the same.

Tim

3:35 am on Feb 19, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 27, 2002
posts:351
votes: 0


Remove the "removethis" from the email address.
8:14 pm on Feb 21, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 27, 2003
posts:1308
votes: 0


Hi Tim,
I wrote, suppose take long to get an answer.
Is there a way to know if you ever been indexed before?
I donīt know if been indexed and dropped, or ever indexed, though before didnīt have stats of site.
This really drives me mad, tried everything, and I donīt have and havnīt ever had any spam on my site, only thing I can imagine is: for years ago I was on an linkfarm, but took away quickly when understood that wasnīt to good, to start with I didnīt know anything so just signed up where I could, and maybe Inktomi ever goes back to check if situation changed.
Just checked the linkfarms and can see sites on there listings that are indexed by Inktomi, so that shouldnīt be the reason either.
Thanks in advance
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members