Forum Moderators: open
This happens to very different kind of pages: frame ones with optimized no frame area, dynamic ones and also classical html pages.
Some of my pages are still normal.
Does anybody have an idea what is this?
If you have a robots.txt file, I'd suggest reviewing it, and validating it here [searchengineworld.com].
Your description matches the symptoms of Google's treatment of a page which has incoming links, but is Disallowed in robots.txt
Of course, it could be a different problem, but that is something to check.
Jim
Do you think one would be now necessary? Or could the problem be an other one?
No. It is OK to not have a robots.txt file.
Another meaning of the situation you describe is that Google has found links to those pages, but has not yet followed the links to index the pages. Therefore, it knows the pages are there, but does not yet know what the titles or descriptions are.
Jim
How long does it take Google to put everything back again?
That's not good...
I can't say for sure... If you are lucky, Googlebot will come back and notice the change in robots.txt and update the data it is preparing for the next update (I don't know if it can or does actually work this way). Otherwise, that "Disallow: /" may just cause Google to drop the whole site during the next update. The site would likely come back at the beginning of February, but that's no consolation...
This may not help, but it may be worth a try:
Before you go kick your developer, let me just say that this accident happens quite often in a development/production environment. Using a formal release checklist for site updates is helpful: Check critical file permissions, .httpd.conf, .htaccess, and robots.txt for correct file versions, content, and file permissions as applicable. Don't make this list too long, though, or it won't be used.
Jim
As soon as I fixed the robots.txt, Googlebot began spidering; actualy every day until Dec. 27th (since then no clue from Googlebot, but it might be normal at this time of the month).
The main page and a few other ones (those known by freshbot) have now their own titles and descriptions back, instead of DMOZīones.
Question Nr. 1: everything thatīs beeing spidered by Googlebot returns a 304 Code. Should I change something in these pages? Or will Google restore the old stage anyway?
Question Nr. 2: I would like Googlebot to visit deeper other pages. My plan is to put these links on those pages which are beeing spidered quite often, but these pages are frame pages. Do you think internal links in the "no frame" area could harm?
Thanks in advance for the answer(s)...
After changing something on every page, "304s" turned into "200s" and now the pages are coming back the one after the other, each day more, with their descriptions and titles.
