Welcome to WebmasterWorld Guest from 54.82.10.219

Forum Moderators: Ocean10000 & phranque

Message Too Old, No Replies

x-robots-tag nofollow in .htaccess

How to test nofollow directive

     
10:49 pm on Jun 22, 2016 (gmt 0)

Junior Member

5+ Year Member Top Contributors Of The Month

joined:Mar 16, 2012
posts: 132
votes: 0


I built some tube sites and I don't want search engines following any links in the player or source files. I thought I'd use something like this but not sure how to test it...

<FilesMatch "\.(flv|gif|jpg|mpeg|mp4|png|swf)$">
Header set X-Robots-Tag "nofollow"
</FilesMatch>
8:27 am on June 23, 2016 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


Better to just put the code in the individual directories:

Header set X-Robots-Tag "nofollow"

However, this does not stop "search engines following any links in the player or source files" only from indexing them and not all support it anyway.
8:40 am on June 23, 2016 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member andy_langton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 27, 2003
posts: 3332
votes: 140


You can test it (if you mean making sure it is set correctly) either with a tool that checks server HTTP response headers, or, depending on your browser, in its developer console. In many cases, hit F12 and you'll see a console. You should be able to select files requested and look at response headers. In Chrome for instance, these are all under a "Network" tab with the option to view headers.
12:51 pm on June 23, 2016 (gmt 0)

Junior Member

5+ Year Member Top Contributors Of The Month

joined:Mar 16, 2012
posts: 132
votes: 0


1) I want that rule to be recursive

2) Response headers show nofollow for site/favicon.png, site/dir/test.png, site/dir/test.mp4 .... (redbot dot org)

I only care about google as far as search engines go. I posted this question to the google section originally.

I ran the ninja bot and didn't see any mention of external links followed from within the player. I'll go to wmt's and check on the test robots page later.

I think that might be a real test. One that says, "Google didn't follow that".
1:04 pm on June 23, 2016 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


Sounds like you set the header field sucessfully.

Google does support the directive (one of the few) but as I said, it doesn't really stop Googlebot from actually "following" the path to the file. The name is misleading. Google will just not attribute the link... which is how I should have explained it earlier :)

It needs to follow the link to get to your server to get the header directive.
2:33 pm on June 23, 2016 (gmt 0)

Junior Member

5+ Year Member Top Contributors Of The Month

joined:Mar 16, 2012
posts: 132
votes: 0


"links in the player or source files"

I know how to control google as far as "my" files. Everything "in the player" is coming from different servers. These are tube sites with embedded video posts. I don't even know how google-bot "sees" these posts. I guess I could make a page with just a player in it and spider-view it.

This is, like most things I want to know, a matter of asking the right question. Which I haven't.
3:02 pm on June 23, 2016 (gmt 0)

Junior Member

5+ Year Member Top Contributors Of The Month

joined:Mar 16, 2012
posts: 132
votes: 0


I forgot, you guys have some tools now. I ran the "On-Page SEO Optimization Analysis Tool" and it reports no external links to source video or images.

with or without robots rules
6:00 pm on June 23, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15313
votes: 708


Response headers show nofollow for site/favicon.png, site/dir/test.png, site/dir/test.mp4

Inquiring minds want to know: What kind of links are embedded in your site's favicon.png? My concern is that the nofollow is attached to the wrong file.

But I have to re-stress what keyplyr said above: "nofollow" doesn't mean "pretend you haven't seen this link". It only means "don't tell them I sent you"* or, more accurately, "don't tell your algorithm I sent you".


* I should probably stop using this wording, because it sounds--wrongly--as if it's got something to do with the Referer header.
12:07 am on June 24, 2016 (gmt 0)

Junior Member

5+ Year Member Top Contributors Of The Month

joined:Mar 16, 2012
posts: 132
votes: 0


"What kind of links are embedded in your site's favicon.png?"

none. Checking header response

Google told me nofollow meant don't follow
12:17 am on June 24, 2016 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


"Google told me nofollow meant don't follow"
False... explained above.

Googles does folllow all links. It uses this data for a multitude of purposes. The tag just requests that the link is not attributed to your site's index profile (ranking) and other SEO.
1:41 am on June 24, 2016 (gmt 0)

Junior Member

5+ Year Member Top Contributors Of The Month

joined:Mar 16, 2012
posts: 132
votes: 0


" 'Nofollow' provides a way for webmasters to tell search engines "Don't follow links on this page" or "Don't follow this specific link."

Not that that's all there is to it. But it's the only way I know of to manage how google handles external links

I have had 404's in wmt's for video files. My own files. So they are getting looked at.
1:54 am on June 24, 2016 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


That's my point. These are the only tools Google affords us but they often do not work the way the webmaster wants. Bottom line: Google (and other SEs) will always do what favors them not us.
3:33 am on June 24, 2016 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


"I have had 404's in wmt's for video files. My own files"

But that has nothing to do with nofollow tags. If Google is reporting a 404 that means for some reason Googlebot could not access that file, usually a poorly written path if the file actually exists.
12:32 pm on June 24, 2016 (gmt 0)

Junior Member

5+ Year Member Top Contributors Of The Month

joined:Mar 16, 2012
posts: 132
votes: 0


My question was if anyone knew a good way to test it
12:52 pm on June 24, 2016 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


My question was if anyone knew a good way to test it
...and Andy Langton & I answered you. It is only possible to test & verify that your code is working to set the header field properly and that Google sees it. You've accomplished that... well done :)

Again, it is not possible to know with 100% accuracy what Google does with that request and as mentioned, you'll never know what *else* Google does with the data found in your links, nofollowed or not.
1:29 am on June 26, 2016 (gmt 0)

Junior Member

5+ Year Member Top Contributors Of The Month

joined:Mar 16, 2012
posts: 132
votes: 0


This is what I was initially looking for, when I began this X-Robots deal, as far as what google-bot can read and follow

support(dot)google(dot)com/webmasters/answer/72746?hl=en
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members