what level do the spiders search to?

how many levels will be spidered

[james gulliver]

I am currently updating a site and have found listings for some of the pages in the search engines but not all pages are listed and some of the pages i have deisgned specifically to pick up traffic are not.

the reason for this seems to be that they are not linked from the home page (third level pages). do spiders stop at the second level during large updates and then spider deeper when they have some spare time, so to speak?

I was under the impression a site would always get spidered to the third level, which explains the need for a sitemap to ensure all pages are searched.

[agerhart]

Search engine spiders will definitely spider down the third, fourth, and levels beyond. It is better to keep your pages closer to the root, but they will still be spidered in deeper directories.

There are many other variables that you should check to make sure your website is completely accessible by the search engine spiders and bots.

[james gulliver]

>There are many other variables that you should check to make sure your website is completely accessible by the search engine spiders and bots.

such as?

is there a resource to check i havent done something that leave a bad taste in the spiders mouths?

[agerhart]

The following things can be detrimental to search engine spidering and access to pages:

- query strings and special characters in URLs
- image maps
- JS navigation
- passwords
- messy code or site structure
- broken links
- frames
- Flash (certain search engines will follow Flash links)

Search Engine World has a good tool to see what the search engine spiders see: Search Engine Spider Simulator [searchengineworld.com]

[Harley_m]

as long as you have a clear comprehensive sitemap, then bots will always be one link away from your root directory...

you can even have an noindex,follow tag if you dont want it human seen...