homepage Welcome to WebmasterWorld Guest from 54.227.34.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Slowing Slurp
An overwhelming responce from Slurp
Hope

10+ Year Member



 
Msg#: 1032 posted 1:26 pm on Oct 16, 2000 (gmt 0)

This may sound like a stupid question, but I am gonna ask anyway. I was always told the only stupid questions are the ones you don't ask.

How do I slow slurp down? Our entire website is spidered every week. That is over 5000 page views a week by slurp for the last year or so. We do update our site regularly, but not the entire site every week. I really don't see a need to have the entire site spidered that often.

I know that I can add slurp to the robots.txt, but that would block it. That is not what I want. I just want it to come once a month and spider the whole site, or something like that.

Any suggestions would be greatly appreciated.

 

stcrim

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1032 posted 1:38 pm on Oct 16, 2000 (gmt 0)

Hope,

Many people are running into the same slurping situation. You are wise to not want to run robots.txt to slow'er down. At least for us, there has been no benefit, no random pages getting indexed, no improvement in ranking - in fact, it appears slup is just visiting with no purpose (can't be, right?)

We have come to ignore the visits and filter for them in the logs. If there's a better suggestion - I too will look forward to it.

Steve

mark roach

10+ Year Member



 
Msg#: 1032 posted 2:02 pm on Oct 16, 2000 (gmt 0)

Sorry to trivialise a serious thread. But when I was at school we used to put small spiders inside the gas nozzles in the science labs. That used to slow them down, they used to walk around dazed for a few seconds before scurrying off.

Alternatively you could detour slurp over to my site, I haven't seen him in ages. Perhaps my past cruelties have finally caught up with me :)

drbill

10+ Year Member



 
Msg#: 1032 posted 2:54 pm on Oct 16, 2000 (gmt 0)

Hope,

I have been getting hit about 12000 a day by slurp that is total on my server. I, like stcrim have just ignored him now as it seems to do not good to get excited about him any more.. Maybe one day I can get excited when i see him but not right now.

Any one getting listings higher than 100 in one word searchs?


Hope

10+ Year Member



 
Msg#: 1032 posted 5:37 pm on Oct 16, 2000 (gmt 0)

>>>it appears slup is just visiting with no purpose<<<

I can't complain about what slurp is picking up. We are lucky to have new pages added to the index all the time.

>>>Alternatively you could detour slurp over to my site, I haven't seen him in ages<<<

mark, I would be happy to send slurp your way if I knew how.

>>>Any one getting listings higher than 100 in one word searchs?<<<

We have lots of listings in the top 10 for one word searches. :)

stcrim

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1032 posted 5:47 pm on Oct 16, 2000 (gmt 0)

>>We have lots of listings in the top 10 for one word searches.

Hope,

Is that new pages or pages pre-June or this year??? And are they competitive keywords and phrases???

Steve

engine

WebmasterWorld Administrator engine us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 1032 posted 5:49 pm on Oct 16, 2000 (gmt 0)

>We are lucky to have new pages added to the index all the time.
Hope, are you doing anything other than just providing a link to these "new" pages?

Hope

10+ Year Member



 
Msg#: 1032 posted 9:15 pm on Oct 16, 2000 (gmt 0)

>>>>Is that new pages or pages pre-June or this year??? And are they competitive keywords and phrases???<<<<

The pages in the top 10 are both old and new. I can't say that they are competetive words.

>>>>are you doing anything other than just providing a link to these "new" pages? <<<<

These pages are just linked from other sections of the site. There is nothing special about them. To be honest, most of these sites only have titles; no description or keywords. They are not optimized in any way. The pages are rich in content and the site is very theme oriented.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved