homepage Welcome to WebmasterWorld Guest from 54.227.141.230
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderators: Brett Tabke

Paid Inclusion Engines and Topics Forum

  posting off  
heavy slurp action?
anyone experiencing it?
kpaul




msg:28889
 2:55 pm on Jul 22, 2003 (gmt 0)

Last couple days I've seen a dramatic increase in spidering from Inktomi. Anyone else noticing this?

thanks.

 

Imaster




msg:28890
 9:59 pm on Jul 22, 2003 (gmt 0)

I noticed that Ink has been doing more frequent crawling on my sites these days.

mcavic




msg:28891
 4:47 am on Jul 25, 2003 (gmt 0)

Yes, very heavy crawling every day lately. But has anyone had large numbers of pages actually added to the index?

martyt




msg:28892
 2:36 pm on Jul 25, 2003 (gmt 0)

I've had over 500 hits from Slurp this month.

But I only have about 48 pages on my entire web site.

Slurp/cat seems to be staying out of the areas restricted by robots.txt and my <meta> tags, so that's not the problem (if they crawled every "add to cart" link, my server would have crashed by now...)

I'm at an absolute loss to explain why they need to hit every page an average of 10 times -- it's pulled my default home page 31 times since July 11. It shows a history of hitting the *exact* same page multiple times within a couple of minutes.

I haven't added any significant new content in the last month and nearly all of my pages were already in the index.

Additionally, it appears Slurp is continuing to hit a number of pages for which I have a 301 permanent redirect in place and have had it there for about 6 weeks.

My guess is that Slurp is somewhat confused this month and isn't keeping track of where it's been.

MWpro




msg:28893
 6:45 pm on Jul 28, 2003 (gmt 0)

A message board one of my clients run has had 4,642 accesses from slurp last week... and there definitely isn't nearly that many topics on it (probably like only 50 topics in all.)

Do you think it's time to block this bot from those pages?

bakedjake




msg:28894
 10:44 pm on Jul 28, 2003 (gmt 0)

Yes, slurp is going nuts over here.

For example, 1338 page requests in the last 24 hours, and 5770 page requests in the last week on one of our e-commerce sites. One thing I've noticed is that it seems to be going deeper into the site than ever before... requesting almost every page on the site, including just about every form of valid dynamic parameter that is linked from somewhere else.

About 20% of the requests over the last week are duplicate.

No signs of additional pages to the index. I'll continue watching this very closely.

dragonlady7




msg:28895
 1:12 am on Jul 29, 2003 (gmt 0)

I have to admit I just clicked on this topic to read it because I thought the title sounded humorously naughty.
^.^
I've noticed Slurp appears in my logs an incredible amount, even though we get zero, and I mean zero, referrals from Inktomi. It's just always been that way. Don't tell me it's getting worse...

kpaul




msg:28896
 1:35 am on Jul 29, 2003 (gmt 0)

maybe they're trying to duplicate google's 'continuous update' but don't have it quite figured out yet? ;)

it doesn't seem too smart as it repeatedly requests the same files on the same days 1000s of them. ugh.

i may wait a while before i block it or restrict it, though, just to see what happens later this year w/Yahoo ...

WileECoyote




msg:28897
 12:12 am on Jul 31, 2003 (gmt 0)

dragonlady7 --> FYI...you won't receive any referals from Inktomi, they don't host their own front-end.

kevinpate




msg:28898
 12:19 am on Jul 31, 2003 (gmt 0)

Unless I'm mistaken, slurp/ink/heavygrabber is the sole reason for our pages to come up in the serps for the M$N crowd. I don't mind the bot traffic as a result, though i do wish that slurp could figure out just what a 301 redirect and a 404 error code really do mean.

Sometimes slurp seems to think I'm just a big kidder and comes back and back and back for non-existent files. Big G's processing of 301 and 404 codes has ink beat by miles and miles from my perspective.

onedumbear




msg:28899
 12:37 am on Aug 1, 2003 (gmt 0)

I don't see "heavy" slurp action because my sites are small. I do notice slurp has spidered all my pages and i have not submitted any through the paid inclusion process. Slurp started visiting me about 3 weeks ago and comes by about every other day. The pages it has requested still do not appear in MSN or elsewhere though.
IMHO there will be a big update soon and hopefully a lot of happy webmasters with new pages added. We can't all be happy though, can we, someone's gotta be on top.
One more interesting note is that today, July 31, about 5 pm... I noticed slurp and googlebots visiting me only 2 or 4 minutes apart from each other.

martyt




msg:28900
 1:10 am on Aug 1, 2003 (gmt 0)

Well, slurp registered a whopping 627 hits on the 57 pages in my web site this month.

Yeah, that's well over 10 hits per page.

And no, it doesn't include hits on robots.txt.

Seems like a pretty poor way to run a spider to me. At very least, there were 470 hits that could have been directed to some other site...

panic




msg:28901
 8:26 pm on Aug 1, 2003 (gmt 0)

I just started on a new site, and it somehow got indexed by Inktomi even though I haven't submitted anywhere, nor am I linked anywhere.

-panic

awicked




msg:28902
 2:40 pm on Aug 3, 2003 (gmt 0)

Inktomi has just finished slurping at my site, any idea how long before the results show in there SERP's?

panic




msg:28903
 5:29 pm on Aug 3, 2003 (gmt 0)

I was indexed the day after I was slurped.

-panic

awicked




msg:28904
 7:02 pm on Aug 3, 2003 (gmt 0)

Panic,

Was that after a deep crawl?

Thanks

panic




msg:28905
 7:24 pm on Aug 4, 2003 (gmt 0)

No. It was a one page deal, since my site's under construction for the time being.

-panic

mcavic




msg:28906
 3:52 pm on Aug 12, 2003 (gmt 0)

Hey - after tons and tons of crawling, Inktomi (just yesterday) has started listing a few of my dynamic pages for the first time.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved