Google pulls title and snippet from DMOZ?

Why and when does it do that?

[genem]

We had a site redesign and all of a sudden, for the home page Google now lists title and description from DMOZ.

I understand that this may have happened when the server was down, but it's now been over 1,5 weeks and the site is very established (used to be PR6 now PR5). All other pages are listed as expected.

Now, here is when it gets really interesting. The same exact thing happens for Yahoo and MSN. Checked server headers - page gets HTTP/1.x 200 OK.

Can anyone shed some light on what is going on here? Where should I look?

[netmeg]

Add this meta tag and it will go away:

<meta name="robots" content="ALL,NOODP,NOYDIR">

(the NOYDIR prevents Yahoo from doing the same thing with the Yahoo Directory)

[genem]

So there is nothing wrong with the server headers? The other thing that makes me nervous is that none of the engines show the crawl date or cache.

Why would they pull DMOZ data for the index page anyway?

[mattg3]

If they don't have enough for their wee description part in their SERPS.

For example: If you enter something short into meta description aka "widget"

Search for widgets;

1. Wikipedia Widget
We are the authority on the galaxy the universe and all
beyond
2. Widget ACME Inc.
Your desrciption bloerp

G pulls an older description from DMOZ if its deemed sufficient.

Just for the Google users that don't go to WP. :)

They spotted such a user in Northern Nebraska .. last month on the 1st of April. The poor user was short sighted though and had a shake in the mouse finger. Needless to say he was immediately sectioned for not clicking on WP. :\

[genem]

Thanks,
But in my scenario all previous meta tags are still there with enough information to feed the bots. Don't see what could have triggered the change...

[mattg3]

Well I have in the moment 3 versions floating about.

Last year I had it on a server where it showed another server of mine (none in sitemaps, analytics). At least they keep tabs on you. That G microphone in that adsense pressie might have more sinister intentions. ;)

[genem]

Well I figured it out.

Apparently, during the upgrade, the old robots.txt has been overwritten and the new one reads:
User-agent:*
Disallow: /

The robots.txt was blocking the crawlers.

The lesson: next time you see your main page listed with DMOZ information and without cache and crawl date - check your robots.txt

Now, most of the important pages are still in the index in both Y! and Google. What would you advise to remedy the situation besides deleting robots.txt and submitting the urls indvidual to the SEs?

[netmeg]

You don't need to resubmit them. Fix your robots.txt and your pages should be crawled again before too long.

[genem]

I just deleted robots.txt, is this ok? Many people suggest that its better to have it, but since I don't have anything to hide, i don't need it right?

[Quadrille]

The main reason (but not the only one), for showing ODP tags in place of yours, is that the search term appears in the ODP description - but not in yours.

If you search for a phrase from your desc meta tag, you'll normally see that appear.

So Google is trying to help; the rationale is that the ODP description is more relevant to the searcher's needs.

You can 'block it', as mentioned above - but it may be worth reviewing your meta description; Google's only trying to increase YOUR relevancy to searchers.

[netmeg]

but since I don't have anything to hide, i don't need it right?

As long as you don't have any rogue weird bots scraping your site or sucking up all your bandwidth, probably not.

[Robert Charlton]

I just deleted robots.txt, is this ok?

Actually, if you want to allow everything, it's a better idea to have either an empty robots.txt file or a robots.txt file that allows everything than it is not to have a robots.txt file at all.

Jim Morgan explains the choices in his final post on this thread....

Google and having *no* robots.txt file
could this be hurting your site?
[webmasterworld.com...]

[genem]

Thanks, Robert.