Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

I cannot understand noindex robot and site: operator

         

serenoo

9:31 am on Feb 18, 2011 (gmt 0)

10+ Year Member



My site lost a lot of google visitors from Feb 2011. So I am analyzing it.
Executing the site operator I found many pages that I blocked by robots.txt. So I went on google webmaster tool removing them one by one. But I discovered that they had
<meta name="robots" content="noindex, nofollow">
so I am wonder why google show them in my page list.
So I searched for words that appear in my noindex pages and I did not find the pages on google.
That means now google shows noindex pages by site operator, but do not show them in index?
I am really confused and worried to see all those pages in my list.
Can someone explain better the situation?

tedster

2:21 pm on Feb 18, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The site: operator results can display any URL Google knows about - sometimes even a URL that is prohibited by robots.txt but has a link pointing to it from somewhere.

In the case of a noindex robots meta tag, googlebot needs to crawl the URL in order to read the meta tag in the first place. So they definitely know about that older version of the page. The way you ended up handling it, requesting removal in Webmaster Tools should be effective.

serenoo

2:33 pm on Feb 18, 2011 (gmt 0)

10+ Year Member



Does site operator show noindex and nofollow urls too?

goodroi

3:21 pm on Feb 21, 2011 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



The site operator can show noindex & nofollow urls.

If Google sees a link pointing to a url they will record the url in their database. Google will not visit the url. They may include a url only listing in site: searches. It will be without title & snippet since it was not crawled.

When using a nofollow tag, please remember that it is possible for another site to link to those pages and not have a nofollow tag.

Please note I say Google "can" and "may" display these urls. Google site: searches are not the most stable search results. It is possible but not guaranteed for these urls to appear in site: searches.

TheMadScientist

3:31 pm on Feb 21, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Executing the site operator I found many pages that I blocked by robots.txt.

That's why you found them...

They didn't visit the page, so they don't know it's not supposed to be indexed. They listened to your robots.txt, but found links to them, so they show the URL in the index.

If you remove the block from the robots.txt so the page is spidered, then you will not see them when you do a site: search, even if there are links to them.

You can really only use one or the other effectively, either robots.txt or noindex. Using them both means the robots.txt takes precedence, so the URL is indexed.

The site operator can show noindex & nofollow urls.

I haven't seen that... Is it new?

ADDED: Oh, maybe you mean when it's on pages that are blocked with robots.txt?