Spiders Not Liking Subfolders?

Am I right or not?

[Kate82]

Okay, so I am a baby of sorts in this industry, but I have worked in SEM for about 2 years. I have never heard of spiders not crawling a site due to subfolders. Am I wrong on this one?

My boss said something about our content not being crawled fully because most of our content is in /pages/

Therefore many of the URLs look like (and this is not a real URL)

www.nameofcompany.com/pages/filename.htm

Is he right? Should I move all my content to the root directory?

[jimbeetle]

No, you shouldn't have to move pages to your root directory. But, if spiders aren't visiting all of your pages some steps you should take are:

Make sure you have a robots.txt file and it is valid. The SearchEngineWorld robots.txt validator page [searchengineworld.com] has links to additional information.

Check that pages don't have inadvertent robots noindex meta tags.

Make sure that pages are well-linked so spiders can find them. Consider using a site map to help them along the trail.

Also, if you have any sort of bot blocking going on in your .htaccess file check that it isn't keeping the good guys out.

Try changing the content a bit on some of the pages. Some bots do a conditional GETs and won't retrieve not modified pages. Similarly, try adding a few new links to your index page, bots love to travel new paths.

And, of course, try to pick up some external links to those interior pages.

[Kate82]

Awesome, yeah, I have checked for all of those things, and our site is indexed as far as I can tell. I just wanted to make sure I wasn't nuts before I went to talk to him. Thanks!