Welcome to WebmasterWorld Guest from 50.19.190.144

Forum Moderators: open

Message Too Old, No Replies

heavy slurp action?

anyone experiencing it?

     
2:55 pm on Jul 22, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Jan 10, 2003
posts:318
votes: 0


Last couple days I've seen a dramatic increase in spidering from Inktomi. Anyone else noticing this?

thanks.

9:59 pm on July 22, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 3, 2003
posts:960
votes: 0


I noticed that Ink has been doing more frequent crawling on my sites these days.
4:47 am on July 25, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Yes, very heavy crawling every day lately. But has anyone had large numbers of pages actually added to the index?
2:36 pm on July 25, 2003 (gmt 0)

Full Member

10+ Year Member

joined:May 29, 2003
posts:202
votes: 0


I've had over 500 hits from Slurp this month.

But I only have about 48 pages on my entire web site.

Slurp/cat seems to be staying out of the areas restricted by robots.txt and my <meta> tags, so that's not the problem (if they crawled every "add to cart" link, my server would have crashed by now...)

I'm at an absolute loss to explain why they need to hit every page an average of 10 times -- it's pulled my default home page 31 times since July 11. It shows a history of hitting the *exact* same page multiple times within a couple of minutes.

I haven't added any significant new content in the last month and nearly all of my pages were already in the index.

Additionally, it appears Slurp is continuing to hit a number of pages for which I have a 301 permanent redirect in place and have had it there for about 6 weeks.

My guess is that Slurp is somewhat confused this month and isn't keeping track of where it's been.

6:45 pm on July 28, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 29, 2002
posts:533
votes: 0


A message board one of my clients run has had 4,642 accesses from slurp last week... and there definitely isn't nearly that many topics on it (probably like only 50 topics in all.)

Do you think it's time to block this bot from those pages?

10:44 pm on July 28, 2003 (gmt 0)

Administrator

WebmasterWorld Administrator bakedjake is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 8, 2003
posts:3783
votes: 1


Yes, slurp is going nuts over here.

For example, 1338 page requests in the last 24 hours, and 5770 page requests in the last week on one of our e-commerce sites. One thing I've noticed is that it seems to be going deeper into the site than ever before... requesting almost every page on the site, including just about every form of valid dynamic parameter that is linked from somewhere else.

About 20% of the requests over the last week are duplicate.

No signs of additional pages to the index. I'll continue watching this very closely.

1:12 am on July 29, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 18, 2003
posts:400
votes: 0


I have to admit I just clicked on this topic to read it because I thought the title sounded humorously naughty.
^.^
I've noticed Slurp appears in my logs an incredible amount, even though we get zero, and I mean zero, referrals from Inktomi. It's just always been that way. Don't tell me it's getting worse...
1:35 am on July 29, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Jan 10, 2003
posts:318
votes: 0


maybe they're trying to duplicate google's 'continuous update' but don't have it quite figured out yet? ;)

it doesn't seem too smart as it repeatedly requests the same files on the same days 1000s of them. ugh.

i may wait a while before i block it or restrict it, though, just to see what happens later this year w/Yahoo ...

12:12 am on July 31, 2003 (gmt 0)

New User

10+ Year Member

joined:Apr 17, 2003
posts:7
votes: 0


dragonlady7 --> FYI...you won't receive any referals from Inktomi, they don't host their own front-end.
12:19 am on July 31, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 2, 2002
posts:1167
votes: 0


Unless I'm mistaken, slurp/ink/heavygrabber is the sole reason for our pages to come up in the serps for the M$N crowd. I don't mind the bot traffic as a result, though i do wish that slurp could figure out just what a 301 redirect and a 404 error code really do mean.

Sometimes slurp seems to think I'm just a big kidder and comes back and back and back for non-existent files. Big G's processing of 301 and 404 codes has ink beat by miles and miles from my perspective.

12:37 am on Aug 1, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 18, 2003
posts:167
votes: 0


I don't see "heavy" slurp action because my sites are small. I do notice slurp has spidered all my pages and i have not submitted any through the paid inclusion process. Slurp started visiting me about 3 weeks ago and comes by about every other day. The pages it has requested still do not appear in MSN or elsewhere though.
IMHO there will be a big update soon and hopefully a lot of happy webmasters with new pages added. We can't all be happy though, can we, someone's gotta be on top.
One more interesting note is that today, July 31, about 5 pm... I noticed slurp and googlebots visiting me only 2 or 4 minutes apart from each other.
1:10 am on Aug 1, 2003 (gmt 0)

Full Member

10+ Year Member

joined:May 29, 2003
posts:202
votes: 0


Well, slurp registered a whopping 627 hits on the 57 pages in my web site this month.

Yeah, that's well over 10 hits per page.

And no, it doesn't include hits on robots.txt.

Seems like a pretty poor way to run a spider to me. At very least, there were 470 hits that could have been directed to some other site...

8:26 pm on Aug 1, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 16, 2003
posts:633
votes: 0


I just started on a new site, and it somehow got indexed by Inktomi even though I haven't submitted anywhere, nor am I linked anywhere.

-panic

2:40 pm on Aug 3, 2003 (gmt 0)

New User

10+ Year Member

joined:July 27, 2003
posts:3
votes: 0


Inktomi has just finished slurping at my site, any idea how long before the results show in there SERP's?
5:29 pm on Aug 3, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 16, 2003
posts:633
votes: 0


I was indexed the day after I was slurped.

-panic

7:02 pm on Aug 3, 2003 (gmt 0)

New User

10+ Year Member

joined:July 27, 2003
posts:3
votes: 0


Panic,

Was that after a deep crawl?

Thanks

7:24 pm on Aug 4, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 16, 2003
posts:633
votes: 0


No. It was a one page deal, since my site's under construction for the time being.

-panic

3:52 pm on Aug 12, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Hey - after tons and tons of crawling, Inktomi (just yesterday) has started listing a few of my dynamic pages for the first time.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members