homepage Welcome to WebmasterWorld Guest from 107.20.34.144
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google shows robots.txt message in place of snippet
sunnyujjawal



 
Msg#: 4485397 posted 9:35 am on Aug 16, 2012 (gmt 0)

For one of my new site i use
<meta name='robots' content='noindex,nofollow' />
it was done 30 days before
but today when i was searching my sites in google with site: operator
for every page in palce of description this message was displayed:

A description for this result is not available because of this site's robots.txt learn more


Learn more was linked to [support.google.com ]
Screenshot is here [imgur.com ].

Can't understand if meta robots are nofollow why Google index these pages.

 

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4485397 posted 9:45 am on Aug 16, 2012 (gmt 0)

You see it in the index because there are links to the page somewhere or other. Google of course needs to crawl the page in order to even read the robots meta tag, but that tag would prevent them from indexing the content of the page in order to generate a snippet.

However, it also sounds from this message like you have a Disallow rule for those URLs in your robots-txt file - is that the case?

rooo

5+ Year Member



 
Msg#: 4485397 posted 10:27 am on Aug 16, 2012 (gmt 0)

I'm seeing something similar too. Months ago we added a robots.txt that blocks everything on a .co.uk domain. Bit stranger though is that we 301 redirect the .co.uk domain to a .com domain.

Seems weird to even show the result - they seem to be ignoring the 301 in this scenario.

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4485397 posted 10:51 am on Aug 16, 2012 (gmt 0)

@rooo - I assume that result is only showing for a site: operator, right? Not sending actual search traffic for a regular search term?

[edited by: tedster at 2:58 pm (utc) on Aug 16, 2012]

rooo

5+ Year Member



 
Msg#: 4485397 posted 12:38 pm on Aug 16, 2012 (gmt 0)

It appears for a brand name search - domains are same as brand name.

aakk9999

WebmasterWorld Administrator 5+ Year Member



 
Msg#: 4485397 posted 1:04 pm on Aug 16, 2012 (gmt 0)

@rooo
You should remove robots.txt Disallow for co.uk domain if you are also redirecting co.uk domain to .com domain as Google will not be able to see and process the redirect if it is not allowed to request the page.

Once Google processes redirects, the co.uk domain will dissapear from Google (this may take some months depending on the number of pages).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved