Forum Moderators: open
Since the sites were all created in a similar way I am now investigating the possibility that the Googlebot is being blocked (or perhaps discouraged!) by the presence of the Dreamweaver inserted text betwen the <html> and <head> tags. This is as follows ...
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><!-- InstanceBegin template="/Templates/widgets.dwt" codeOutsideHTMLIsLocked="false" -->
<head>
Can anyone offer any advice or assistance with this?
You say that your site isn't being spidered - do you mean that Googlebot is not accessing the pages in question at all? If so, it's not going to be an on-page factor.
Have you validated your robots.txt [searchengineworld.com]? If your pages are being spidered, have you validated the html? Is there anything else on the page which would invite a penalty - some (how shall I put it...) over-zealous SEO, for example?
I have tried all the usual validations, etc. but everything seems to be fine. The main site in question was spidered for over two years before this happened. And there is nothing in there that is spammy. Anyway, thanks for the comments. It looks like this comment thingy is not my problem so I'll have to look elsewhere I suppose.
Those tags are only used by dreamweaver, They become comment tags when it comes to browsers,
So yes you should look somewhere else to see potential problems,
Recently Googlebot is a bit slow in picking up new pages, If your pages are pretty new then you have to wait for moretime to get the pages into google index,