homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

Why not spidered?

5+ Year Member

Msg#: 2892 posted 7:55 pm on Apr 11, 2006 (gmt 0)

Hi, I'm very very new to SEO. I have a site that is not getting spidered, except for the main page. I added a site map, what else can i do? I'm at a loss here.

Sorry to ask such a silly question.



WebmasterWorld Senior Member 10+ Year Member

Msg#: 2892 posted 4:02 pm on Apr 12, 2006 (gmt 0)

The spiders will sooner or later get to the rest of your web site. Get a few web sites to link to your other web pages on your web site to speed it up the process of spidering the whole web site.


10+ Year Member

Msg#: 2892 posted 7:45 pm on Apr 12, 2006 (gmt 0)

google sitemaps is also pretty usefull, it might jumpstart the process. But yeah, in the beginning sometimes things can move slowly, but google is just trying gauge how serious you are about your site. just keep adding content and things will fall into place. also, if you don't have a robots.txt file, you can add this to your meta tags to let googlebot know you want to be spidered:



5+ Year Member

Msg#: 2892 posted 1:50 am on Apr 13, 2006 (gmt 0)

Hello, jtaylor.
I would like to share with you this quote from one of Google's engineers: "One of the classic crawling strategies that Google has used is the amount of PageRank on your pages. So just because your site has been around for a couple years (or that you submit a sitemap), that doesn’t mean that we’ll automatically crawl every page on your site. In general, getting good quality links would probably help us know to crawl your site more deeply." As you can see, according to this, your site may be waiting for two years and still not be spidered if it has no incoming links. As robotsdobetter said, you have better get some links to speed the process.
Good luck.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved