Slowing Slurp

An overwhelming responce from Slurp

[Hope]

This may sound like a stupid question, but I am gonna ask anyway. I was always told the only stupid questions are the ones you don't ask.

How do I slow slurp down? Our entire website is spidered every week. That is over 5000 page views a week by slurp for the last year or so. We do update our site regularly, but not the entire site every week. I really don't see a need to have the entire site spidered that often.

I know that I can add slurp to the robots.txt, but that would block it. That is not what I want. I just want it to come once a month and spider the whole site, or something like that.

Any suggestions would be greatly appreciated.

[stcrim]

Hope,

Many people are running into the same slurping situation. You are wise to not want to run robots.txt to slow'er down. At least for us, there has been no benefit, no random pages getting indexed, no improvement in ranking - in fact, it appears slup is just visiting with no purpose (can't be, right?)

We have come to ignore the visits and filter for them in the logs. If there's a better suggestion - I too will look forward to it.

Steve

[mark roach]

Sorry to trivialise a serious thread. But when I was at school we used to put small spiders inside the gas nozzles in the science labs. That used to slow them down, they used to walk around dazed for a few seconds before scurrying off.

Alternatively you could detour slurp over to my site, I haven't seen him in ages. Perhaps my past cruelties have finally caught up with me :)

[drbill]

Hope,

I have been getting hit about 12000 a day by slurp that is total on my server. I, like stcrim have just ignored him now as it seems to do not good to get excited about him any more.. Maybe one day I can get excited when i see him but not right now.

Any one getting listings higher than 100 in one word searchs?

[Hope]

>>>it appears slup is just visiting with no purpose<<<

I can't complain about what slurp is picking up. We are lucky to have new pages added to the index all the time.

>>>Alternatively you could detour slurp over to my site, I haven't seen him in ages<<<

mark, I would be happy to send slurp your way if I knew how.

>>>Any one getting listings higher than 100 in one word searchs?<<<

We have lots of listings in the top 10 for one word searches. :)

[stcrim]

>>We have lots of listings in the top 10 for one word searches.

Hope,

Is that new pages or pages pre-June or this year??? And are they competitive keywords and phrases???

Steve

[engine]

>We are lucky to have new pages added to the index all the time.
Hope, are you doing anything other than just providing a link to these "new" pages?

[Hope]

>>>>Is that new pages or pages pre-June or this year??? And are they competitive keywords and phrases???<<<<

The pages in the top 10 are both old and new. I can't say that they are competetive words.

>>>>are you doing anything other than just providing a link to these "new" pages? <<<<

These pages are just linked from other sections of the site. There is nothing special about them. To be honest, most of these sites only have titles; no description or keywords. They are not optimized in any way. The pages are rich in content and the site is very theme oriented.