homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Gold Sponsor 2015!
Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque

Webmaster General Forum

Checklist for low spidering
Main points to visit when you have this issue

10+ Year Member

Msg#: 11904 posted 7:45 am on Jun 6, 2006 (gmt 0)

Hi all ... I have recently taken over the maintenance of two new sites and they were redesigned and moved onto a new server. The issue I am having is that Google is simply not spidering the sites after the server move. Previously they were spidered daily and on a big scale.

We have over 50 sites on the server and two of them in question who are big blue chip companies are getting visited by Google once a week whilst other smaller sites on the server which have NO updates at all get loads of visits from the Google bot. So I had to ask myself WHY!?

Here is a checklist for anybody else who may suffer a similar problem:

1) We have a firewall on the server
2) The sites robots.txt file in messed up
3) The server ip has been assoaciated with spam
4) The server company is blocking spiders

Now these are the main ones we could think of and I would appreciated any further points to be added.



10+ Year Member

Msg#: 11904 posted 3:37 pm on Jun 6, 2006 (gmt 0)

You could submit the site map for Google to come along and spider it, also have you tried removing the robots.txt?

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved