homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Google pulls title and snippet from DMOZ?
Why and when does it do that?

 3:26 pm on May 14, 2007 (gmt 0)

We had a site redesign and all of a sudden, for the home page Google now lists title and description from DMOZ.

I understand that this may have happened when the server was down, but it's now been over 1,5 weeks and the site is very established (used to be PR6 now PR5). All other pages are listed as expected.

Now, here is when it gets really interesting. The same exact thing happens for Yahoo and MSN. Checked server headers - page gets HTTP/1.x 200 OK.

Can anyone shed some light on what is going on here? Where should I look?



 8:29 pm on May 14, 2007 (gmt 0)

Add this meta tag and it will go away:

<meta name="robots" content="ALL,NOODP,NOYDIR">

(the NOYDIR prevents Yahoo from doing the same thing with the Yahoo Directory)


 8:52 am on May 15, 2007 (gmt 0)

So there is nothing wrong with the server headers? The other thing that makes me nervous is that none of the engines show the crawl date or cache.

Why would they pull DMOZ data for the index page anyway?


 10:49 am on May 15, 2007 (gmt 0)

If they don't have enough for their wee description part in their SERPS.

For example: If you enter something short into meta description aka "widget"

Search for widgets;

1. Wikipedia Widget
We are the authority on the galaxy the universe and all
2. Widget ACME Inc.
Your desrciption bloerp

G pulls an older description from DMOZ if its deemed sufficient.

Just for the Google users that don't go to WP. :)

They spotted such a user in Northern Nebraska .. last month on the 1st of April. The poor user was short sighted though and had a shake in the mouse finger. Needless to say he was immediately sectioned for not clicking on WP. :\


 1:16 pm on May 15, 2007 (gmt 0)

But in my scenario all previous meta tags are still there with enough information to feed the bots. Don't see what could have triggered the change...


 3:17 pm on May 15, 2007 (gmt 0)

Well I have in the moment 3 versions floating about.

Last year I had it on a server where it showed another server of mine (none in sitemaps, analytics). At least they keep tabs on you. That G microphone in that adsense pressie might have more sinister intentions. ;)


 3:56 pm on May 15, 2007 (gmt 0)

Well I figured it out.

Apparently, during the upgrade, the old robots.txt has been overwritten and the new one reads:
Disallow: /

The robots.txt was blocking the crawlers.

The lesson: next time you see your main page listed with DMOZ information and without cache and crawl date - check your robots.txt

Now, most of the important pages are still in the index in both Y! and Google. What would you advise to remedy the situation besides deleting robots.txt and submitting the urls indvidual to the SEs?


 4:16 pm on May 15, 2007 (gmt 0)

You don't need to resubmit them. Fix your robots.txt and your pages should be crawled again before too long.


 8:04 am on May 16, 2007 (gmt 0)

I just deleted robots.txt, is this ok? Many people suggest that its better to have it, but since I don't have anything to hide, i don't need it right?


 10:03 am on May 16, 2007 (gmt 0)

The main reason (but not the only one), for showing ODP tags in place of yours, is that the search term appears in the ODP description - but not in yours.

If you search for a phrase from your desc meta tag, you'll normally see that appear.

So Google is trying to help; the rationale is that the ODP description is more relevant to the searcher's needs.

You can 'block it', as mentioned above - but it may be worth reviewing your meta description; Google's only trying to increase YOUR relevancy to searchers.


 2:53 pm on May 16, 2007 (gmt 0)

but since I don't have anything to hide, i don't need it right?

As long as you don't have any rogue weird bots scraping your site or sucking up all your bandwidth, probably not.

Robert Charlton

 6:18 am on May 22, 2007 (gmt 0)

I just deleted robots.txt, is this ok?

Actually, if you want to allow everything, it's a better idea to have either an empty robots.txt file or a robots.txt file that allows everything than it is not to have a robots.txt file at all.

Jim Morgan explains the choices in his final post on this thread....

Google and having *no* robots.txt file
could this be hurting your site?


 8:21 am on May 22, 2007 (gmt 0)

Thanks, Robert.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved