I have noticed for the last few days that some results in the SERPS are saying...
A description for this result is not available because of this site's robots.txt – learn more The "learn more" part is a hyperlink that takes you to [support.google.com...]
This text is showing in place of the page description.
tedster
9:48 pm on Aug 31, 2012 (gmt 0)
Another member mentioned this a couple week's ago, too. I haven't run into it myself, but I'd guess it's an extension of showing URLs that are Disallowed in robots.txt for which they can't collect enough data to put together a reasonable snippet. Why such a URL might rank is another question, however.
Leosghost
9:55 pm on Aug 31, 2012 (gmt 0)
Been seeing this regularly too, frequently for sites at #1, #2 or #3 for search terms..
Lame_Wolf
10:05 pm on Aug 31, 2012 (gmt 0)
Tedster, I have seen it on a number of sites. I found it this time whilst googling for an Adsense problem. If you Google, "failed to render creative" - with quotation marks, you will see an example of it.
Leosghost
10:16 pm on Aug 31, 2012 (gmt 0)
Interestingly that example leads to a login page which has nothing to do with the search term..although the "term" is in the link in SERP..
Is G indexing behind password protected logins and then including the links in SERPS ?
If the relevant content is behind the logins, how does G know ..toolbars, Chrome ?
Lame_Wolf
10:25 pm on Aug 31, 2012 (gmt 0)
And what use is the link (to the robots.txt) that Google supplies? It is of no use to the user, only the site owner.
lucy24
1:15 am on Sep 1, 2012 (gmt 0)
Is G indexing behind password protected logins and then including the links in SERPS ?
Totally possible. I've seen questions here about how to set up a "Satisfy any" directive so the site can be seen by logged-in users BUT ALSO by search engines. And what about things like those blasted JSTOR links where the robot has obvious read the whole article, because not all the search terms are in the part the general public is allowed to see? The search engine only knows that it can get there. It doesn't know that not everyone else can.
I haven't run into it myself,
Do you mean by accident? It's trivial to find it on purpose. Constrain the search to your site, and do an exact-phrase search for link text that leads to a roboted-out page. Best if the phrase occurs on no more than 1000 pages in the whole site, probably.
I don't know what the thinking is. Either that the site has so many members, they might as well let the whole thing show up in SERPs-- or you're supposed to be so tantalized by the search-result snippet that you rush right over and sign up.
