Welcome to WebmasterWorld Guest from 22.214.171.124
I understand that this may have happened when the server was down, but it's now been over 1,5 weeks and the site is very established (used to be PR6 now PR5). All other pages are listed as expected.
Now, here is when it gets really interesting. The same exact thing happens for Yahoo and MSN. Checked server headers - page gets HTTP/1.x 200 OK.
Can anyone shed some light on what is going on here? Where should I look?
For example: If you enter something short into meta description aka "widget"
Search for widgets;
1. Wikipedia Widget
We are the authority on the galaxy the universe and all
2. Widget ACME Inc.
Your desrciption bloerp
G pulls an older description from DMOZ if its deemed sufficient.
Just for the Google users that don't go to WP. :)
They spotted such a user in Northern Nebraska .. last month on the 1st of April. The poor user was short sighted though and had a shake in the mouse finger. Needless to say he was immediately sectioned for not clicking on WP. :\
Apparently, during the upgrade, the old robots.txt has been overwritten and the new one reads:
The robots.txt was blocking the crawlers.
The lesson: next time you see your main page listed with DMOZ information and without cache and crawl date - check your robots.txt
Now, most of the important pages are still in the index in both Y! and Google. What would you advise to remedy the situation besides deleting robots.txt and submitting the urls indvidual to the SEs?
If you search for a phrase from your desc meta tag, you'll normally see that appear.
So Google is trying to help; the rationale is that the ODP description is more relevant to the searcher's needs.
You can 'block it', as mentioned above - but it may be worth reviewing your meta description; Google's only trying to increase YOUR relevancy to searchers.
I just deleted robots.txt, is this ok?
Actually, if you want to allow everything, it's a better idea to have either an empty robots.txt file or a robots.txt file that allows everything than it is not to have a robots.txt file at all.
Jim Morgan explains the choices in his final post on this thread....
Google and having *no* robots.txt file
could this be hurting your site?