Internal linking to pages blocking Google with robots.txt?

[richinberlin]

I've always assumed that linking internally to pages blocked by robots.txt was bad... so I don't do it and recommend my clients don't do it.

I assume that link juice passes to the page, but since Google is not indexing the target page, they can't then see where its to be distributed, so it's lost.

Anyone think differently?

[edited by: Robert_Charlton at 9:35 am (utc) on Feb 23, 2020]
[edit reason] Added text lost when we dropped description line... [/edit]

[lammert]

Basic question: Are those links to blocked pages helpful for the visitor? If not, then you should delete those internal blocked pages altogether IMO as they neither have value for the visitor, nor for the search engines. But if they are useful for the visitor there is no reason not to link to them.

[richinberlin]

They are useful, but there is no reason to link to them per se and every reason not to let them be indexed. They are alternate language versions of the same site, auto-translated. They could just as easily be only accessed by a form submission which took the user to the alternate language version but didn't allow Google to follow as not a link.

So they are useful, but Google says don't put up auto-translate pages to be indexed.

So, back to the actual question. Does linking to pages blocked from google crawling, bleed link juice? I say yes, who says otherwise?

[Kendo]

As I see it, it is unavoidable. A lot of pages will be internally linked for feedback, support, login, password retrieval and online ordering, not to mention pages that may not be mobile friendly but necessary, that should not be indexed.

Or have I missed something?

[richinberlin]

IN this case, 2/3rds of the site is banned from being indexed. So the point is if its 6 pages of 100,000 then its less of a concern for lost link juice if its 60,000 of 90,000.

So I still need the actual question answered. Sorry to be direct, but I don't need to debate the when, or why etc, just the if

[lucy24]

Links are for humans. They are secondarily useful for search engines, but you should be thinking first about whether a human will benefit.

[iamlost]

What lucy24 said x100.

My sites have roughly 50% of pages SE blocked without noticeable effect on traffic (I track SE referral traffic not mythical rankings).

I first started blocking internally (I redirect any external direct links) linked pages about 15years ago and it rose quickly to ~40% and has been slowly increasing since.

As is often the case it’s not so much what you do but how.

[richinberlin]

Ok. So. That's awesome guys. Blocking parts of your website from robots can have a positive effect. Yep. I get that. Am doing it. The question is does linking to these blocked parts lose link juice?

And yes links are for people... And for search engines. Yes I do get this.

But does anyone have a position on the actual question I've asked? Rather than a philosophical position about links generally.

If I link to pages blocked by robots is it going to have the same effect as a noindexed link and please... Let's not go down the new rules for noindex rabbit hole.

If I link to such a page... Is it like linking to a 404 page?

[lammert]

The answers you have been given are not philosophical. They are an integral part of how the Google algorithm for distributing link juice now works.

Ten years ago your question would have been valid. Link juice and the distribution of it was merely a mathematical equation. But after a lot of abuse of links, Google has adjusted to a method where it may take the actual human usage of links into account. Blocking a link in robots.txt is still a signal, but it won't guarantee the outcome of the link juice distribution pattern for your site.

[richinberlin]

Ok. I'm going to accept the consensus answer of we don't know. Google may do this or that

Thanks

[Andy Langton]

My experience is that link value flows to the excluded pages. You can see the effect of this when blocked pages still rank, just without a snippet. Google even reports those pages in Search Console ("Indexed, though blocked by robots.txt" - [support.google.com...]

Of course, link value can't flow from those pages, which is the theoretically lost link value from excluding (as well as the fact that the excluded page itself won't rank). I only use robots.txt when the inclusion in search results causes worse problems than the lost theoretical value.

[Kendo]

I learned a long time ago not to use Chrome for testing pages on a site that is not yet ready for the public. Because the pages would be indexed simply by visiting them in Chrome. I don't know if that is still the case or not because I learnt my lesson and have not done it since.

[richinberlin]

Thanks Andy. Mirrors my position. I'm not after formulas, guarantees etc. I spend a lot of time studying big, successful sites. Its obvious to me the ones that rank the best are merciless in squirrelling away link juice. They also do everything else right as well... but I figure its got to be a factor. I was just curious if anyone had an actual scientific test done to prove it.

I am 100% white hat in my SEO. Great content, honest links, best practice structured markup, fast websites... but on top of that ensuring link juice go to the most important pages is my goal. It's not funnelling to not link to 404 pages, and my position is its also not link funnelling to make sure links don't got to other pages where link juice dissipates.

Anyway, thanks I have what I need.

[lammert]

Anyway, thanks I have what I need.

That's obvious. And what do you give the community in return? That no-one gave you the hard answer directly doesn't mean no-one has the answers. But those who have the knowledge may be wary to share it in an open setting like this board.

[richinberlin]

That's obvious. And what do you give the community in return? That no-one gave you the hard answer directly doesn't mean no-one has the answers. But those who have the knowledge may be wary to share it in an open setting like this board.

Hundreds of posts under another account name attached to an email I no longer use. Member since 2003. I stopped visiting much some years ago when as I saw it, the waters became too mixed. Those able to swim, those wanting to shout they were good at swimming without actually showing they were in the water. I get some people are all about the community... I'm more about the asking and providing of useful answers. Ive sure provided my share of both.

Seems you are saying I have to show I am also contributing to get a straight, useful, accurate answer from someone who knows what they are on about? That you and others were deliberately vague to hide knowledge for your own reasons? Seems odd.

[lammert]

Not necessarily so strict. What I tried to say is that the answers you get may not only depend on what you ask, but also on how you ask it and the amount of engagement you show either inside or outside the topic. To make it look less than a one-way information stream. But maybe I interpreted some wordings you used in the replies in this topic in the wrong way. No need to first engage in a number of other topics before being taken seriously :)

There are many knowledgeable members who are mostly in read-only mode but step in now and then to give their really valuable insights. Especially on such a topic where deep knowledge and often a lot of testing and experience is needed as your particular issue.

[apriljesser]

If you are linking your blog pages internally then don't block them through robots, as if you are putting your keyword in blog and linking that keyword to their related url page then it will be beneficial for your website. It will help in ranking as well as driving traffic to the website.

[Parth_Shah]

Internal linking is for the users to easily navigate to your website. So if you internal link a page that is blocked by robots.txt it's not a big of a problem as eventually the page is going to open. The main reason of using robots.txt is to let the crawlers know that they don't need to index the page so that it doesn't appear on SERPs.

[NickMNS]

First one thing needs to be made clear. Robots.txt does not prevent Google from indexing a page it prevents Google from crawling a page.

As to the question of the loss of link juice. No, these are all internal links it has no impact on link juice. So its moot. The only impact it could have is on the pages linked from the blocked page if those links are the only links pointing to those pages. In other words you could inadvertently be creating orphan pages. Otherwise, the link juice is redistributed through the other links of the website.

I would think that there are much more effective places to be spending time to improve rankings than to worry about these things.