homepage Welcome to WebmasterWorld Guest from 54.205.207.53
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
heavy slurp action?
anyone experiencing it?
kpaul

10+ Year Member



 
Msg#: 2228 posted 2:55 pm on Jul 22, 2003 (gmt 0)

Last couple days I've seen a dramatic increase in spidering from Inktomi. Anyone else noticing this?

thanks.

 

Imaster

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2228 posted 9:59 pm on Jul 22, 2003 (gmt 0)

I noticed that Ink has been doing more frequent crawling on my sites these days.

mcavic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2228 posted 4:47 am on Jul 25, 2003 (gmt 0)

Yes, very heavy crawling every day lately. But has anyone had large numbers of pages actually added to the index?

martyt

10+ Year Member



 
Msg#: 2228 posted 2:36 pm on Jul 25, 2003 (gmt 0)

I've had over 500 hits from Slurp this month.

But I only have about 48 pages on my entire web site.

Slurp/cat seems to be staying out of the areas restricted by robots.txt and my <meta> tags, so that's not the problem (if they crawled every "add to cart" link, my server would have crashed by now...)

I'm at an absolute loss to explain why they need to hit every page an average of 10 times -- it's pulled my default home page 31 times since July 11. It shows a history of hitting the *exact* same page multiple times within a couple of minutes.

I haven't added any significant new content in the last month and nearly all of my pages were already in the index.

Additionally, it appears Slurp is continuing to hit a number of pages for which I have a 301 permanent redirect in place and have had it there for about 6 weeks.

My guess is that Slurp is somewhat confused this month and isn't keeping track of where it's been.

MWpro

10+ Year Member



 
Msg#: 2228 posted 6:45 pm on Jul 28, 2003 (gmt 0)

A message board one of my clients run has had 4,642 accesses from slurp last week... and there definitely isn't nearly that many topics on it (probably like only 50 topics in all.)

Do you think it's time to block this bot from those pages?

bakedjake

WebmasterWorld Administrator bakedjake us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2228 posted 10:44 pm on Jul 28, 2003 (gmt 0)

Yes, slurp is going nuts over here.

For example, 1338 page requests in the last 24 hours, and 5770 page requests in the last week on one of our e-commerce sites. One thing I've noticed is that it seems to be going deeper into the site than ever before... requesting almost every page on the site, including just about every form of valid dynamic parameter that is linked from somewhere else.

About 20% of the requests over the last week are duplicate.

No signs of additional pages to the index. I'll continue watching this very closely.

dragonlady7

10+ Year Member



 
Msg#: 2228 posted 1:12 am on Jul 29, 2003 (gmt 0)

I have to admit I just clicked on this topic to read it because I thought the title sounded humorously naughty.
^.^
I've noticed Slurp appears in my logs an incredible amount, even though we get zero, and I mean zero, referrals from Inktomi. It's just always been that way. Don't tell me it's getting worse...

kpaul

10+ Year Member



 
Msg#: 2228 posted 1:35 am on Jul 29, 2003 (gmt 0)

maybe they're trying to duplicate google's 'continuous update' but don't have it quite figured out yet? ;)

it doesn't seem too smart as it repeatedly requests the same files on the same days 1000s of them. ugh.

i may wait a while before i block it or restrict it, though, just to see what happens later this year w/Yahoo ...

WileECoyote

10+ Year Member



 
Msg#: 2228 posted 12:12 am on Jul 31, 2003 (gmt 0)

dragonlady7 --> FYI...you won't receive any referals from Inktomi, they don't host their own front-end.

kevinpate

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2228 posted 12:19 am on Jul 31, 2003 (gmt 0)

Unless I'm mistaken, slurp/ink/heavygrabber is the sole reason for our pages to come up in the serps for the M$N crowd. I don't mind the bot traffic as a result, though i do wish that slurp could figure out just what a 301 redirect and a 404 error code really do mean.

Sometimes slurp seems to think I'm just a big kidder and comes back and back and back for non-existent files. Big G's processing of 301 and 404 codes has ink beat by miles and miles from my perspective.

onedumbear

10+ Year Member



 
Msg#: 2228 posted 12:37 am on Aug 1, 2003 (gmt 0)

I don't see "heavy" slurp action because my sites are small. I do notice slurp has spidered all my pages and i have not submitted any through the paid inclusion process. Slurp started visiting me about 3 weeks ago and comes by about every other day. The pages it has requested still do not appear in MSN or elsewhere though.
IMHO there will be a big update soon and hopefully a lot of happy webmasters with new pages added. We can't all be happy though, can we, someone's gotta be on top.
One more interesting note is that today, July 31, about 5 pm... I noticed slurp and googlebots visiting me only 2 or 4 minutes apart from each other.

martyt

10+ Year Member



 
Msg#: 2228 posted 1:10 am on Aug 1, 2003 (gmt 0)

Well, slurp registered a whopping 627 hits on the 57 pages in my web site this month.

Yeah, that's well over 10 hits per page.

And no, it doesn't include hits on robots.txt.

Seems like a pretty poor way to run a spider to me. At very least, there were 470 hits that could have been directed to some other site...

panic

10+ Year Member



 
Msg#: 2228 posted 8:26 pm on Aug 1, 2003 (gmt 0)

I just started on a new site, and it somehow got indexed by Inktomi even though I haven't submitted anywhere, nor am I linked anywhere.

-panic

awicked

10+ Year Member



 
Msg#: 2228 posted 2:40 pm on Aug 3, 2003 (gmt 0)

Inktomi has just finished slurping at my site, any idea how long before the results show in there SERP's?

panic

10+ Year Member



 
Msg#: 2228 posted 5:29 pm on Aug 3, 2003 (gmt 0)

I was indexed the day after I was slurped.

-panic

awicked

10+ Year Member



 
Msg#: 2228 posted 7:02 pm on Aug 3, 2003 (gmt 0)

Panic,

Was that after a deep crawl?

Thanks

panic

10+ Year Member



 
Msg#: 2228 posted 7:24 pm on Aug 4, 2003 (gmt 0)

No. It was a one page deal, since my site's under construction for the time being.

-panic

mcavic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2228 posted 3:52 pm on Aug 12, 2003 (gmt 0)

Hey - after tons and tons of crawling, Inktomi (just yesterday) has started listing a few of my dynamic pages for the first time.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved