x-robots-tag nofollow in .htaccess

How to test nofollow directive

[dougwilson]

I built some tube sites and I don't want search engines following any links in the player or source files. I thought I'd use something like this but not sure how to test it...

<FilesMatch "\.(flv|gif|jpg|mpeg|mp4|png|swf)$">
Header set X-Robots-Tag "nofollow"
</FilesMatch>

[keyplyr]

Better to just put the code in the individual directories:

Header set X-Robots-Tag "nofollow"

However, this does not stop "search engines following any links in the player or source files" only from indexing them and not all support it anyway.

[Andy Langton]

You can test it (if you mean making sure it is set correctly) either with a tool that checks server HTTP response headers, or, depending on your browser, in its developer console. In many cases, hit F12 and you'll see a console. You should be able to select files requested and look at response headers. In Chrome for instance, these are all under a "Network" tab with the option to view headers.

[dougwilson]

1) I want that rule to be recursive

2) Response headers show nofollow for site/favicon.png, site/dir/test.png, site/dir/test.mp4 .... (redbot dot org)

I only care about google as far as search engines go. I posted this question to the google section originally.

I ran the ninja bot and didn't see any mention of external links followed from within the player. I'll go to wmt's and check on the test robots page later.

I think that might be a real test. One that says, "Google didn't follow that".

[keyplyr]

Sounds like you set the header field sucessfully.

Google does support the directive (one of the few) but as I said, it doesn't really stop Googlebot from actually "following" the path to the file. The name is misleading. Google will just not attribute the link... which is how I should have explained it earlier :)

It needs to follow the link to get to your server to get the header directive.

[dougwilson]

"links in the player or source files"

I know how to control google as far as "my" files. Everything "in the player" is coming from different servers. These are tube sites with embedded video posts. I don't even know how google-bot "sees" these posts. I guess I could make a page with just a player in it and spider-view it.

This is, like most things I want to know, a matter of asking the right question. Which I haven't.

[dougwilson]

I forgot, you guys have some tools now. I ran the "On-Page SEO Optimization Analysis Tool" and it reports no external links to source video or images.

with or without robots rules

[lucy24]

Response headers show nofollow for site/favicon.png, site/dir/test.png, site/dir/test.mp4

Inquiring minds want to know: What kind of links are embedded in your site's favicon.png? My concern is that the nofollow is attached to the wrong file.

But I have to re-stress what keyplyr said above: "nofollow" doesn't mean "pretend you haven't seen this link". It only means "don't tell them I sent you"* or, more accurately, "don't tell your algorithm I sent you".


* I should probably stop using this wording, because it sounds--wrongly--as if it's got something to do with the Referer header.

[dougwilson]

"What kind of links are embedded in your site's favicon.png?"

none. Checking header response

Google told me nofollow meant don't follow

[keyplyr]

"Google told me nofollow meant don't follow"
False... explained above.

Googles does folllow all links. It uses this data for a multitude of purposes. The tag just requests that the link is not attributed to your site's index profile (ranking) and other SEO.

[dougwilson]

" 'Nofollow' provides a way for webmasters to tell search engines "Don't follow links on this page" or "Don't follow this specific link."

Not that that's all there is to it. But it's the only way I know of to manage how google handles external links

I have had 404's in wmt's for video files. My own files. So they are getting looked at.

[keyplyr]

That's my point. These are the only tools Google affords us but they often do not work the way the webmaster wants. Bottom line: Google (and other SEs) will always do what favors them not us.

[keyplyr]

"I have had 404's in wmt's for video files. My own files"

But that has nothing to do with nofollow tags. If Google is reporting a 404 that means for some reason Googlebot could not access that file, usually a poorly written path if the file actually exists.

[dougwilson]

My question was if anyone knew a good way to test it

[keyplyr]

My question was if anyone knew a good way to test it

...and Andy Langton & I answered you. It is only possible to test & verify that your code is working to set the header field properly and that Google sees it. You've accomplished that... well done :)

Again, it is not possible to know with 100% accuracy what Google does with that request and as mentioned, you'll never know what *else* Google does with the data found in your links, nofollowed or not.

[dougwilson]

This is what I was initially looking for, when I began this X-Robots deal, as far as what google-bot can read and follow

support(dot)google(dot)com/webmasters/answer/72746?hl=en