Forum Moderators: open
Before I panic and spend days updating client sites. How important are doctype and encoding statements (and we might as well include robot.txt files) for Google (and other se’s)?
My gut is that yes they may be important for various reasons, but I would be surprised to learn that Google may not index pages/sites that don’t use them as they would be skipping a tremendous amount of existing web pages.
The !DOCTYPE should have no effect on Google spidering, unless you are using some advanced XML perhaps.
However, I would include those tags, and run the code through the HTML validtor just to make sure that there are no tag typos in the code.
To answer the question that he did not address: robots.txt has just one function: keeping robots out of your site. So the lack of a robots.txt file is certainly not going to prevent indexing. A badly constructed one certainly may do so.
In another thread I was told that Google is not indexing pages from one of my client sites because it does not include doctype statements or encoding statements.
I find that very hard to believe.
I would suggest setting up a very basic HTML site map and link to it via the home page. Normally works wonders!
Google indexes what it can "see", so missing closing tags, and badly nested stuff can make it "see" something other than what was intended.
Try a Google Search for exactly this: meta name="description" content= to "see" what I mean.
