homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

Googlebot dilemma
Crawling hard - Site not found

10+ Year Member

Msg#: 26965 posted 7:20 pm on Dec 4, 2004 (gmt 0)

Googlebots (the moz compatible ones) have been hitting a site of mine VERY hard for the last two weeks (13,000 pages in 5 hours yesterday, 10,000 in 3 hours today), this site is, and has been missing from the index since August 2004. site:site.co.uk produces no results, but allinurl:site.co.uk produces 21,500 pages all of which appear to be the dreaded pagejacker sites, with clickclocker.php in their urls.

Should I ban googlebot from the site, or should I allow it, in the hope that whatever caused/is causing the sites demise, will be rectified by google, sooner rather than later? (bandwidth isn't an issue, but other websites on the same server are suffering degraded performance whilst this heavy crawling is going on)


Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved