Welcome to WebmasterWorld Guest from 54.166.220.138

Message Too Old, No Replies

Google pulls title and snippet from DMOZ?

Why and when does it do that?

     
3:26 pm on May 14, 2007 (gmt 0)

5+ Year Member



We had a site redesign and all of a sudden, for the home page Google now lists title and description from DMOZ.

I understand that this may have happened when the server was down, but it's now been over 1,5 weeks and the site is very established (used to be PR6 now PR5). All other pages are listed as expected.

Now, here is when it gets really interesting. The same exact thing happens for Yahoo and MSN. Checked server headers - page gets HTTP/1.x 200 OK.

Can anyone shed some light on what is going on here? Where should I look?

8:29 pm on May 14, 2007 (gmt 0)

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Add this meta tag and it will go away:

<meta name="robots" content="ALL,NOODP,NOYDIR">

(the NOYDIR prevents Yahoo from doing the same thing with the Yahoo Directory)

8:52 am on May 15, 2007 (gmt 0)

5+ Year Member



So there is nothing wrong with the server headers? The other thing that makes me nervous is that none of the engines show the crawl date or cache.

Why would they pull DMOZ data for the index page anyway?

10:49 am on May 15, 2007 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



If they don't have enough for their wee description part in their SERPS.

For example: If you enter something short into meta description aka "widget"

Search for widgets;

1. Wikipedia Widget
We are the authority on the galaxy the universe and all
beyond
2. Widget ACME Inc.
Your desrciption bloerp

G pulls an older description from DMOZ if its deemed sufficient.

Just for the Google users that don't go to WP. :)

They spotted such a user in Northern Nebraska .. last month on the 1st of April. The poor user was short sighted though and had a shake in the mouse finger. Needless to say he was immediately sectioned for not clicking on WP. :\

1:16 pm on May 15, 2007 (gmt 0)

5+ Year Member



Thanks,
But in my scenario all previous meta tags are still there with enough information to feed the bots. Don't see what could have triggered the change...
3:17 pm on May 15, 2007 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Well I have in the moment 3 versions floating about.

Last year I had it on a server where it showed another server of mine (none in sitemaps, analytics). At least they keep tabs on you. That G microphone in that adsense pressie might have more sinister intentions. ;)

3:56 pm on May 15, 2007 (gmt 0)

5+ Year Member



Well I figured it out.

Apparently, during the upgrade, the old robots.txt has been overwritten and the new one reads:
User-agent:*
Disallow: /

The robots.txt was blocking the crawlers.

The lesson: next time you see your main page listed with DMOZ information and without cache and crawl date - check your robots.txt

Now, most of the important pages are still in the index in both Y! and Google. What would you advise to remedy the situation besides deleting robots.txt and submitting the urls indvidual to the SEs?

4:16 pm on May 15, 2007 (gmt 0)

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



You don't need to resubmit them. Fix your robots.txt and your pages should be crawled again before too long.
8:04 am on May 16, 2007 (gmt 0)

5+ Year Member



I just deleted robots.txt, is this ok? Many people suggest that its better to have it, but since I don't have anything to hide, i don't need it right?
10:03 am on May 16, 2007 (gmt 0)

WebmasterWorld Senior Member quadrille is a WebmasterWorld Top Contributor of All Time 10+ Year Member



The main reason (but not the only one), for showing ODP tags in place of yours, is that the search term appears in the ODP description - but not in yours.

If you search for a phrase from your desc meta tag, you'll normally see that appear.

So Google is trying to help; the rationale is that the ODP description is more relevant to the searcher's needs.

You can 'block it', as mentioned above - but it may be worth reviewing your meta description; Google's only trying to increase YOUR relevancy to searchers.

2:53 pm on May 16, 2007 (gmt 0)

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



but since I don't have anything to hide, i don't need it right?

As long as you don't have any rogue weird bots scraping your site or sucking up all your bandwidth, probably not.

6:18 am on May 22, 2007 (gmt 0)

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I just deleted robots.txt, is this ok?

Actually, if you want to allow everything, it's a better idea to have either an empty robots.txt file or a robots.txt file that allows everything than it is not to have a robots.txt file at all.

Jim Morgan explains the choices in his final post on this thread....

Google and having *no* robots.txt file
could this be hurting your site?
[webmasterworld.com...]

8:21 am on May 22, 2007 (gmt 0)

5+ Year Member



Thanks, Robert.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month