Forum Moderators: open
Get some inbound links. Google likes to crawl sites that have inbound links. There is no garantee that it will index and entire site that doesn't have links (pr value).
Other than than, just wait. Eventually googlebot well settle in and grab it all.
Check out Paynt's Getting Started [webmasterworld.com] thread for some more details about finding information here.
Brett is right about getting some incoming links as they will certianly help.
I would also question if it is only Google that has the challenge with deep spidering, or whether you have noticed other spiders sniffing but not coming in. If the latter, then you might want to double check your robots.txt file. First, make sure you have one (I assume you do) and second make sure the syntax is correct. You may think you arfe inviting spiders, but you may be inadvertantly sending them away by mistake.
Use the Search link at the top of the page and search for robots.txt and you should get some good clues on how to fine tune your robots file.
Onya
Woz
The fast crawler also visits my sites and does that same thing as googlebot.
I currently don't have a robots.txt, will creating one help my changes of being indexed?
Theoretically no, but I believe there was a time an engine failed to index some sites because they didn't have a robots.txt. Apparently a programming error on the part of the engine. However, I like to be sure these days and include on even if it is a standard "let every one in" file.
The other advantage of a robots file is being able to restrict parts of the site you don't want indexed. This may not be an issue at the moment, but it may be later on. Now is as good a time as any to start learning about implementing your robots file.
Onya
Woz
Most of my pages have includes (example: <?php include ("menu.html"); ?>, would this have any affect to search engine crawlers?
No. The server will deliver the same content to the spider and the browser (when you don't cloak).
If you worry you can add this line to your root's .htaccess file
AddType application/x-httpd-php .htm
and *.htm files will be parsed for PHP. Just use *.htm instead of *.php.
EX: widgets.com/widgets.asp?type=&
They didnt like to go any further than that. Changed them to following:
EX: widgets.com/widgets.asp?type=black
Does anyone know if the size of a file prevents a crawl?
The domain is new, there is no robots.txt file.
I did not submit the site to google, just let it re-spider the pages with links and find the new site.
