Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: open
How do I slow slurp down? Our entire website is spidered every week. That is over 5000 page views a week by slurp for the last year or so. We do update our site regularly, but not the entire site every week. I really don't see a need to have the entire site spidered that often.
I know that I can add slurp to the robots.txt, but that would block it. That is not what I want. I just want it to come once a month and spider the whole site, or something like that.
Any suggestions would be greatly appreciated.
Many people are running into the same slurping situation. You are wise to not want to run robots.txt to slow'er down. At least for us, there has been no benefit, no random pages getting indexed, no improvement in ranking - in fact, it appears slup is just visiting with no purpose (can't be, right?)
We have come to ignore the visits and filter for them in the logs. If there's a better suggestion - I too will look forward to it.
Alternatively you could detour slurp over to my site, I haven't seen him in ages. Perhaps my past cruelties have finally caught up with me :)
I have been getting hit about 12000 a day by slurp that is total on my server. I, like stcrim have just ignored him now as it seems to do not good to get excited about him any more.. Maybe one day I can get excited when i see him but not right now.
Any one getting listings higher than 100 in one word searchs?
I can't complain about what slurp is picking up. We are lucky to have new pages added to the index all the time.
>>>Alternatively you could detour slurp over to my site, I haven't seen him in ages<<<
mark, I would be happy to send slurp your way if I knew how.
>>>Any one getting listings higher than 100 in one word searchs?<<<
We have lots of listings in the top 10 for one word searches. :)
The pages in the top 10 are both old and new. I can't say that they are competetive words.
>>>>are you doing anything other than just providing a link to these "new" pages? <<<<
These pages are just linked from other sections of the site. There is nothing special about them. To be honest, most of these sites only have titles; no description or keywords. They are not optimized in any way. The pages are rich in content and the site is very theme oriented.