heavy slurp action?

anyone experiencing it?

[kpaul]

Last couple days I've seen a dramatic increase in spidering from Inktomi. Anyone else noticing this?

thanks.

[Imaster]

I noticed that Ink has been doing more frequent crawling on my sites these days.

[mcavic]

Yes, very heavy crawling every day lately. But has anyone had large numbers of pages actually added to the index?

[martyt]

I've had over 500 hits from Slurp this month.

But I only have about 48 pages on my entire web site.

Slurp/cat seems to be staying out of the areas restricted by robots.txt and my <meta> tags, so that's not the problem (if they crawled every "add to cart" link, my server would have crashed by now...)

I'm at an absolute loss to explain why they need to hit every page an average of 10 times -- it's pulled my default home page 31 times since July 11. It shows a history of hitting the *exact* same page multiple times within a couple of minutes.

I haven't added any significant new content in the last month and nearly all of my pages were already in the index.

Additionally, it appears Slurp is continuing to hit a number of pages for which I have a 301 permanent redirect in place and have had it there for about 6 weeks.

My guess is that Slurp is somewhat confused this month and isn't keeping track of where it's been.

[MWpro]

A message board one of my clients run has had 4,642 accesses from slurp last week... and there definitely isn't nearly that many topics on it (probably like only 50 topics in all.)

Do you think it's time to block this bot from those pages?

[bakedjake]

Yes, slurp is going nuts over here.

For example, 1338 page requests in the last 24 hours, and 5770 page requests in the last week on one of our e-commerce sites. One thing I've noticed is that it seems to be going deeper into the site than ever before... requesting almost every page on the site, including just about every form of valid dynamic parameter that is linked from somewhere else.

About 20% of the requests over the last week are duplicate.

No signs of additional pages to the index. I'll continue watching this very closely.

[dragonlady7]

I have to admit I just clicked on this topic to read it because I thought the title sounded humorously naughty.
^.^
I've noticed Slurp appears in my logs an incredible amount, even though we get zero, and I mean zero, referrals from Inktomi. It's just always been that way. Don't tell me it's getting worse...

[kpaul]

maybe they're trying to duplicate google's 'continuous update' but don't have it quite figured out yet? ;)

it doesn't seem too smart as it repeatedly requests the same files on the same days 1000s of them. ugh.

i may wait a while before i block it or restrict it, though, just to see what happens later this year w/Yahoo ...

[WileECoyote]

dragonlady7 --> FYI...you won't receive any referals from Inktomi, they don't host their own front-end.

[kevinpate]

Unless I'm mistaken, slurp/ink/heavygrabber is the sole reason for our pages to come up in the serps for the M$N crowd. I don't mind the bot traffic as a result, though i do wish that slurp could figure out just what a 301 redirect and a 404 error code really do mean.

Sometimes slurp seems to think I'm just a big kidder and comes back and back and back for non-existent files. Big G's processing of 301 and 404 codes has ink beat by miles and miles from my perspective.

[onedumbear]

I don't see "heavy" slurp action because my sites are small. I do notice slurp has spidered all my pages and i have not submitted any through the paid inclusion process. Slurp started visiting me about 3 weeks ago and comes by about every other day. The pages it has requested still do not appear in MSN or elsewhere though.
IMHO there will be a big update soon and hopefully a lot of happy webmasters with new pages added. We can't all be happy though, can we, someone's gotta be on top.
One more interesting note is that today, July 31, about 5 pm... I noticed slurp and googlebots visiting me only 2 or 4 minutes apart from each other.

[martyt]

Well, slurp registered a whopping 627 hits on the 57 pages in my web site this month.

Yeah, that's well over 10 hits per page.

And no, it doesn't include hits on robots.txt.

Seems like a pretty poor way to run a spider to me. At very least, there were 470 hits that could have been directed to some other site...

[panic]

I just started on a new site, and it somehow got indexed by Inktomi even though I haven't submitted anywhere, nor am I linked anywhere.

-panic

[awicked]

Inktomi has just finished slurping at my site, any idea how long before the results show in there SERP's?

[panic]

I was indexed the day after I was slurped.

-panic

[awicked]

Panic,

Was that after a deep crawl?

Thanks

[panic]

No. It was a one page deal, since my site's under construction for the time being.

-panic

[mcavic]

Hey - after tons and tons of crawling, Inktomi (just yesterday) has started listing a few of my dynamic pages for the first time.