homepage Welcome to WebmasterWorld Guest from 54.242.200.172
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Google Hammering My site (1500+ hits already today)
How to edit robots.txt to set delay
wfernley




msg:1526894
 2:36 pm on Aug 24, 2005 (gmt 0)

Hey I heard there is a way you can delay a crawler on your site. Right now Google is hammering my site and has been for the past month. I find it very weird because it keeps searching the same pages over and over.

Does anyone know if I can delay the crawler so it doesn't go so fast?

Thanks :)

Wes

 

Dijkgraaf




msg:1526895
 12:21 am on Aug 25, 2005 (gmt 0)

The following tells the bot to delay 60 seconds
crawl-delay: 60
However this is a non-standard command and not supported by all bots so it should only be used for those user-agents that support it.

Also see [google.com...]

CSE_Monkey




msg:1526896
 4:50 pm on Aug 25, 2005 (gmt 0)

Someone at Google has taken Yahoo!s 20 B index to heart and they have been very aggressive at crawling the last two weeks...at least in these parts.

Cheers!

Gargen




msg:1526897
 6:47 am on Aug 31, 2005 (gmt 0)

i got 50k page views in one day from Gbot it was insane i had to kick the IP range for about an hour because my visitors were having issues

wfernley




msg:1526898
 12:05 pm on Aug 31, 2005 (gmt 0)

I just got a reply back from Google. They said the Googlebot was having trouble because I had php session ids showing in the url. It was causing problems. They provided a link on how to remove it which really helped because I have been wanting to remove it for a while and I didn't know how. It was a problem relating to my PHPBB forums.

Thanks for the relpies

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved