Welcome to WebmasterWorld Guest from 23.20.110.176

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Google pulls title and snippet from DMOZ?

Why and when does it do that?

     
3:26 pm on May 14, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2005
posts:107
votes: 0


We had a site redesign and all of a sudden, for the home page Google now lists title and description from DMOZ.

I understand that this may have happened when the server was down, but it's now been over 1,5 weeks and the site is very established (used to be PR6 now PR5). All other pages are listed as expected.

Now, here is when it gets really interesting. The same exact thing happens for Yahoo and MSN. Checked server headers - page gets HTTP/1.x 200 OK.

Can anyone shed some light on what is going on here? Where should I look?

8:29 pm on May 14, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12905
votes: 194


Add this meta tag and it will go away:

<meta name="robots" content="ALL,NOODP,NOYDIR">

(the NOYDIR prevents Yahoo from doing the same thing with the Yahoo Directory)

8:52 am on May 15, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2005
posts: 107
votes: 0


So there is nothing wrong with the server headers? The other thing that makes me nervous is that none of the engines show the crawl date or cache.

Why would they pull DMOZ data for the index page anyway?

10:49 am on May 15, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 9, 2006
posts:800
votes: 0


If they don't have enough for their wee description part in their SERPS.

For example: If you enter something short into meta description aka "widget"

Search for widgets;

1. Wikipedia Widget
We are the authority on the galaxy the universe and all
beyond
2. Widget ACME Inc.
Your desrciption bloerp

G pulls an older description from DMOZ if its deemed sufficient.

Just for the Google users that don't go to WP. :)

They spotted such a user in Northern Nebraska .. last month on the 1st of April. The poor user was short sighted though and had a shake in the mouse finger. Needless to say he was immediately sectioned for not clicking on WP. :\

1:16 pm on May 15, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2005
posts: 107
votes: 0


Thanks,
But in my scenario all previous meta tags are still there with enough information to feed the bots. Don't see what could have triggered the change...
3:17 pm on May 15, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 9, 2006
posts:800
votes: 0


Well I have in the moment 3 versions floating about.

Last year I had it on a server where it showed another server of mine (none in sitemaps, analytics). At least they keep tabs on you. That G microphone in that adsense pressie might have more sinister intentions. ;)

3:56 pm on May 15, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2005
posts: 107
votes: 0


Well I figured it out.

Apparently, during the upgrade, the old robots.txt has been overwritten and the new one reads:
User-agent:*
Disallow: /

The robots.txt was blocking the crawlers.

The lesson: next time you see your main page listed with DMOZ information and without cache and crawl date - check your robots.txt

Now, most of the important pages are still in the index in both Y! and Google. What would you advise to remedy the situation besides deleting robots.txt and submitting the urls indvidual to the SEs?

4:16 pm on May 15, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12905
votes: 194


You don't need to resubmit them. Fix your robots.txt and your pages should be crawled again before too long.
8:04 am on May 16, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2005
posts: 107
votes: 0


I just deleted robots.txt, is this ok? Many people suggest that its better to have it, but since I don't have anything to hide, i don't need it right?
10:03 am on May 16, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member quadrille is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Feb 22, 2002
posts:3455
votes: 0


The main reason (but not the only one), for showing ODP tags in place of yours, is that the search term appears in the ODP description - but not in yours.

If you search for a phrase from your desc meta tag, you'll normally see that appear.

So Google is trying to help; the rationale is that the ODP description is more relevant to the searcher's needs.

You can 'block it', as mentioned above - but it may be worth reviewing your meta description; Google's only trying to increase YOUR relevancy to searchers.

2:53 pm on May 16, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12905
votes: 194


but since I don't have anything to hide, i don't need it right?

As long as you don't have any rogue weird bots scraping your site or sucking up all your bandwidth, probably not.

6:18 am on May 22, 2007 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:11527
votes: 226


I just deleted robots.txt, is this ok?

Actually, if you want to allow everything, it's a better idea to have either an empty robots.txt file or a robots.txt file that allows everything than it is not to have a robots.txt file at all.

Jim Morgan explains the choices in his final post on this thread....

Google and having *no* robots.txt file
could this be hurting your site?
[webmasterworld.com...]

8:21 am on May 22, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 11, 2005
posts:107
votes: 0


Thanks, Robert.