Welcome to WebmasterWorld Guest from

Forum Moderators: brotherhood of lan & mack

Message Too Old, No Replies

Why not spidered?



7:55 pm on Apr 11, 2006 (gmt 0)

5+ Year Member

Hi, I'm very very new to SEO. I have a site that is not getting spidered, except for the main page. I added a site map, what else can i do? I'm at a loss here.

Sorry to ask such a silly question.


4:02 pm on Apr 12, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

The spiders will sooner or later get to the rest of your web site. Get a few web sites to link to your other web pages on your web site to speed it up the process of spidering the whole web site.


7:45 pm on Apr 12, 2006 (gmt 0)

10+ Year Member

google sitemaps is also pretty usefull, it might jumpstart the process. But yeah, in the beginning sometimes things can move slowly, but google is just trying gauge how serious you are about your site. just keep adding content and things will fall into place. also, if you don't have a robots.txt file, you can add this to your meta tags to let googlebot know you want to be spidered:



1:50 am on Apr 13, 2006 (gmt 0)

5+ Year Member

Hello, jtaylor.
I would like to share with you this quote from one of Google's engineers: "One of the classic crawling strategies that Google has used is the amount of PageRank on your pages. So just because your site has been around for a couple years (or that you submit a sitemap), that doesn’t mean that we’ll automatically crawl every page on your site. In general, getting good quality links would probably help us know to crawl your site more deeply." As you can see, according to this, your site may be waiting for two years and still not be spidered if it has no incoming links. As robotsdobetter said, you have better get some links to speed the process.
Good luck.

Featured Threads

Hot Threads This Week

Hot Threads This Month