Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

Why not spidered?

7:55 pm on Apr 11, 2006 (gmt 0)

New User

10+ Year Member

joined:Mar 21, 2006
votes: 0

Hi, I'm very very new to SEO. I have a site that is not getting spidered, except for the main page. I added a site map, what else can i do? I'm at a loss here.

Sorry to ask such a silly question.

4:02 pm on Apr 12, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 31, 2004
votes: 0

The spiders will sooner or later get to the rest of your web site. Get a few web sites to link to your other web pages on your web site to speed it up the process of spidering the whole web site.
7:45 pm on Apr 12, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 15, 2004
votes: 0

google sitemaps is also pretty usefull, it might jumpstart the process. But yeah, in the beginning sometimes things can move slowly, but google is just trying gauge how serious you are about your site. just keep adding content and things will fall into place. also, if you don't have a robots.txt file, you can add this to your meta tags to let googlebot know you want to be spidered:


1:50 am on Apr 13, 2006 (gmt 0)

New User

10+ Year Member

joined:Apr 8, 2006
votes: 0

Hello, jtaylor.
I would like to share with you this quote from one of Google's engineers: "One of the classic crawling strategies that Google has used is the amount of PageRank on your pages. So just because your site has been around for a couple years (or that you submit a sitemap), that doesn’t mean that we’ll automatically crawl every page on your site. In general, getting good quality links would probably help us know to crawl your site more deeply." As you can see, according to this, your site may be waiting for two years and still not be spidered if it has no incoming links. As robotsdobetter said, you have better get some links to speed the process.
Good luck.