Google no longer indexing content in onLoad hidden sections?

[CaptainSalad2]

It seems google are no longer indexing any content in my “read more” panels that are hidden during the “onLoad” event.

Shame because this is a user friendly way to display content and prevent unnecessary scrolling if the user isn’t interested in certain sections of a page.

On my hobby site I have client profiles that are displayed using an accordion jquery script, about 10 sections worth of content is organised per client in this way to prevent overwhelming the client with endless scrolling. But when I cut and past any section that’s “hidden” on load it is no longer found on google!

They used to index this stuff…

[aristotle]

Is Google's algorithm still taking that "hidden" content into account when determining the overall page's search rankings for its main keywords? Because it wouldn't make sense for the algorithm to disregard that content in its overall evaluation of the page.

[CaptainSalad2]

Very hard to tell, I recently developed a client login so each member can mange their profile content and they have been adding new content/photos/videos (and more) which are on hidden tabs like "Videos", "FAQ's" and stuff like that.

I am not seeing any improvement in rankings for these pages even though they now contain in depth fresh content, this is why I tried copying and pasting some content from hidden panels into google. Even home page hidden content that has been around for 9 months so its not like it just hasn't been indexed yet!

I figured using the onload event so the page content is visible with and without JS enabled Google would index all content (as opposed to the CSS "display:none" approach). Shame because its only hidden to improve the user experience, not to "trick" Google..

Side note BING is still indexing onLoad hidden content! Google DID used to :/

[not2easy]

Google's crawlers can decipher javascript, but cannot click to generate the additional content. There are ways you can present that content to the crawler (this is not something I do, but I read about it so it may or may not be helpful for you.) If your jquery uses AJAX, you can read more here: [developers.google.com...]

[denisl]

So is this the same as my tabbed content where I use javascript to change css display:none to display:block when you click on a tab.

I was rather hoping that Google saw it all.

[aakk9999]

So is this the same as my tabbed content where I use javascript to change css display:none to display:block when you click on a tab.

I was rather hoping that Google saw it all.

It is not quite the same as tabbed content. Google can see display:none as the tab content is already in the page HTML and your javascript just makes it visible on the page.

However, if onclick you use AJAX to get additional content from the server (i.e. it is not already embedded somewhere in page HTML), then the only way Google can index that content is if there is a #! version of URL, which is described in Google guide not2easy has linked to above.

One of ways to verify whether Google sees your content is to do "View source" and search for the content. If it is there, but just hidden via display:none or via some other css trick, then Google can see it and index it.

A care must be taken using display:none however - and this is that there must be an action on the page that makes content visible. If the content is in HTML and hidden from user using display:none and there is no action the user can perform to make this content visible, then this is called cloaking (showing a different content to user than to search engine) and can be penalised by Google.

@CaptainSalad2
What happens after your onLoad event with regards to Read More links? Do you do another request to the server to get this additional content or do you just make content visible, but the content was already part of your page HTML?

Also, you said Google used to index it before - are you sure nothing changed from your side? How are you judging that it is not indexed now and know it was indexed before?

[seoskunk]

Can the OP provide an example of code thats ignored. Lotts of elements of site use javascript to show/hide content, like menu's for instance. I think a code snippit would help everyone.

[CaptainSalad2]

Just to clarify the content is NOT loaded via AJAX (I wanted to go this route but was concerned search engines could not index it).

The page content is part of the HTML but organised into hidden sections on the document.ready event. The content is accessible to the user via navigational tabs that open and close onclick. Without JS enabled all content is visible!

Example of code used:
$(document).ready(function() {$("#Panel").hide();})

Reason for me noticing that google USED to index content in these panels is there was a thread here asking if content is/isn’t indexed a while back so I tested it out myself by cut/paste chunks into Google which was found.

Recently there was a SER article that stated Google was now waiting for the page and all associated scripts to load before indexing the page? This is my reason for rechecking good still indexes my hidden content!

I guess I can write some server side code to detect the google bot and withhold the script which hides the panels to ensure they index the entire pages content?

[CaptainSalad2]

Additional thoughts, its possible many of the users who already have websites are just cutting/pasting content from their own sites and G is ignoring it?

[aakk9999]

This is my reason for rechecking good still indexes my hidden content!

Have you tested it recently? You can use site:example.com in combination with search phrase that is only visible onload and see if Google returns this page.

Something like:

site:example.com "part of my sentence that gets hidden onload but is in HTML"

If you have tested this, let us know the results.

[CaptainSalad2]

Just tried

site:example.com "part of my sentence that doesn't get hidden onload but is in HTML" (result: showed the site + sentence)

Then repeated above with a sentence from a home page hidden panel, doesn't find it...

Home page panel text is unique to my site, not copied from elsewhere..

[levo]

Posted this 2 weeks ago [webmasterworld.com...]

[CaptainSalad2]

levo, I guess that explains it then!

Shame if Google has indeed stopped indexing accessibly hidden content that's in the HTML. So much for build for the user experience, now I have to write a script to block the JQUERY from being sent to the googlebot and ensure its all seen...ugh!

[petehall]

@CaptainSalad2 if I try this my pages are still found by Google but the meta description / listing is not adjusted to show the matching text.

Is this what you are seeing or are you saying the page is not found at all?

[FranticFish]

Do you block Google from your script files? They have recently announced that sites that do this might notice an impact on rankings.

I have to admit, it sounds absolutely bizarre for Google to ignore parts of the HTML text because they can detect the fact that its visibility is being toggled with Javascript that they are not allowed to crawl, but that IS the sort of decision that I can see them making these days.

[lucy24]

Shame if Google has indeed stopped indexing accessibly hidden content that's in the HTML.

Makes perfect sense to me. If you go to a page in response to a search, you expect to find the searched-for content right there in plain sight on the page. Not hidden behind some "more info" link that you might not even happen to notice (and which wouldn't show up with a browser "Find" if your visitor wanted to head straight for their desired text). It's potentially even more annoying than material which turns out to require a login.

[FranticFish]

Yet this is the same company that will say "Did you mean to search for [what you definitely DIDN'T want to search for]", rewrites title tags, and continues to rank pages off links pointing to them over content on the page (ref: 'click here').

The message is 'it's the context, not the exact text, that we're interested in'.

So I think it's a bit bizarre for the same company to decide to refuse to index content that is publicly accessible but can be revealed with a tap, a click, or a mouseover - especially when the same bot does things like input queries into search forms and index the results, if allowed.

I can only hope that it's a minor penalty: a stick to force webmasters to allow script file indexing.

[JD_Toims]

Yet this is the same company that will say...

And has said blocking JS is not an issue, directory submissions are not a problem, etc. but now we're called spammers if we do it, sometimes by the same person who said it was fine and acceptable to do -- Go figure.

[petehall]

It's SO easy to get round though. Instead of tabs etc we go back to the old drawing board of all the text being on the page and jumping up/down the page.

This is not better for users. It really is a step backwards. I think it's amusing personally.

[FranticFish]

That may not be necessary.

I was on a major car manufacturer's site the other day and they use carousels extensively to reveal content when exploring each car. These are in the source code but not visible when the page loads. I've checked and content from these sections is indexed just fine.

They do not block Google from either of the two locations their script files are accessed from.

I also noticed that they load most of their script files just before the </body> tag, and the 'document.ready' function is also right down there after all the content, so the spider finds it last (I'm not a Javascript programmer so I don't know if this is significant but I do usually see these in the <head>, BEFORE content).

Unless there's a whitelist for major brands when it comes to acceptable code, which I consider unlikely, then I think it's worth testing this as there certainly appears to be an acceptable way to reveal content.

I'm assuming that they aren't showing Google a different version of the page :)

[JD_Toims]

<aside>

I'm not a Javascript programmer so I don't know if this is significant but I do usually see these in the <head>, BEFORE content.

JavaScript doesn't care as long as the scripts load in the correct order and nothing "inline" needs a later script to work -- Right before the </body> tag is the correct place to load all scripts for "initial render" speed.

---

Generally when a browser finds a script to load, rather than concurrently downloading elements it only use one connection to the site, which loads the script, will wait until the script is done loading and then parse the script prior to rendering anything else or continuing to load other respurces so it can tell if the script impacts anything.

Not too big of a deal where they are if you only have one or two "light" scripts, but sites using mammoth scripts, large numbers of scripts, etc. will render faster initially by loading those after all graphics and HTML are loaded -- Depending on the connection speed, number of scripts, server speed, etc. the difference can range from, "Meh, that didn't do anything...", to, "Wow! You gotta be kidding me..."

</aside>

[aakk9999]

So I think it's a bit bizarre for the same company to decide to refuse to index content that is publicly accessible but can be revealed with a tap, a click, or a mouseover

I am not doubting what others have said, however I have checked a few sites that have content in tabs or reveal on hover and from what I can see, the initially hidden tab content and the content revealed on mouseover is also indexed despite needing a click/hover to reveal it.

So what is different?

[Broadway]

I posted this issue in October but it didn't gain any traction:
[webmasterworld.com...]

I've tested this a bit since. If I google site:mysite... and specific, unique keywords:

1) Google returns the exact right page that holds the content in a "click to expand" box.

2) What google doesn't do is on the SERP's page. It does not show the exact matching content from the expand box bolded in the SERP's.

So, I think google does index the content, but doesn't give it the same elevated treatment as content that always shows on the page.

I've run this test on expand-content boxes that were created as recently as 11/8/14 and indexed by google on 11/9/14.

[aakk9999]

@broadway, interesting.

I have not searched using "site:" command. I just put some exact text from on-hover box and from the "hidden" tab in the Google search box using quotes.

I got two results:

a) Text that was searched for in quotes from (non-focused) tabs was shown as bolded in SERPs instead of page description

b) Text that was searched for in quotes from the box that opens on hover/mouseover was not shown and meta description was shown instead. However, the correct page that contained this text was identified.

[diddlydazz]

@broadway / aakk9999

i see the same thing, a certain page i have has a paragraph at the top (visible by default) and then several sections hidden by default using display: none - with an image for "show/hide"

doing a site: search for the top paragraph shows the bolded excerpt from the page

however, searching for any text in the "display: none" sections returns the page but shows the meta desc

FWIW,i have not noticed any change in rankings for this page (i changed it around 4 months ago - before the change those specific sections were not "hidden by default")

[FranticFish]

@ JD_Toims - thanks for the explanation. I should have made the connection: as Googlebot IS a basic browser, then the order shouldn't be important. It may not render the page in the traditional sense, but it must be doing something similar to that to understand it. And if it's crawling script files I doubt it cares where in the code it finds them.

[Broadway]

I can concur that I see no traffic (ranking) changes. (although I have no idea when this change first took effect).

doesn't Wikipedia use accordion boxes for mobile?

and from what I see in webmaster tools, doesn't Google render the page in both desktop and mobile formats?

so isn't this a first, google did something that doesn't favor Wikipedia?

[Broadway]

SERoundtable did an update on this story.

As an example of what Google does, Barry pointed out a Google help document that uses click to expand boxes but puts the actual answer on a second page.

I looked at the example given and what it boiled down to was a lot of small pages with 1 to 300 word counts.

So I can either design-user friendly with click to expand boxes and not receive full-fledged indexing.

Or I can do it like Google does and risk a Panda penalty.

[thejosh]

I'm not sure I fully believe what was said in the Hangout.

English Google Webmaster Central office-hours hangout
At around 10:50 into hangout
https://www.youtube.com/watch?v=tFSI4cpJX-I [youtube.com]

Google still uses click-to-expand content (see here for one example [google.com ]) and the content is indexed (grab a snipped of content and run a query).

I do question whether the spiders are looking for display:none or other hard-to-decipher code configurations, but i don't see any direct benefit to Google from doing this.

For example: Remember when they made Analytics search queries "not provided?" That encouraged people to spend more on AdWords. This just seems like it's being blown out of proportion with little benefit. Am I wrong? Do you guys see a point I'm missing?

[edited by: aakk9999 at 2:01 am (utc) on Nov 26, 2014]
[edit reason] Added link to Hangout [/edit]

[aakk9999]

Welcome to WebmasterWorld, thejosh!

Having done some tests, I think it is to do with "bolding" in SERPs snippets, but it appears to have some side effects to rankings. The benefit woud be to the end-user (searcher) - which is important to Google. But I do not think the content is as ignored as mentioned on Hangout. I do think however the ranking is downgraded.

As JohnMu says, it could make sense not to bold the text on the page if the text is in closed tab/expanding section. Because if the user comes to that page and searches for that text - it is not there. Cannot be seen and searching using Ctrl+F on the browser page returns no results, which may be frustrating to the user.

FEW EXAMPLES:
Lets take Google Tags FAQ page you gave as an example http://www.google.com/tagmanager/faq.html [google.com]

If you click on the above link, you can see that the first section is shown in the expanded mode. All others below it are in closed mode.

1) If I search for a text from the first (initially expanded) section:

Tags are tiny bits of website code that let you measure traffic and visitor behavior, understand the impact of online advertising and social channels

then #1 result in SERPs is the above URL and the snippet shows the text I searched for above, bolded. This happens regardless whether I searched for the text in quotes or witout quotes.

2) If I search for a text from one of the closed sections:

Redundant or incorrectly applied tags can distort your measurement and result in duplicate costs or missing data.

If I search for it without quote, it is not #1 result. On google.com the faq URL is #2 result, and the text does not show in snippet. Instead, other sentences from initially visible part of the page show in snippet. Note that #1 result is stackoverflow where searched for sentence is in "visible" part of the page (not in section that opens onclick)

If I search for the above sentence within quotes, then the faq URL is #1, but snippet shows meta description, there is no bolded sentence shown in snippet since it is not directly visible when the page loads (i.e. the sentence is visible only when the expanded section opens).

---

This change *may have been* perhaps implemented as a result of observing Chrome behaviour where visitors are searching for a text, clicking on the result that has this text, but hidden initially, and then using Ctrl+F to find the text on the page - and not finding it. Maybe this is why Google decided not to bold the text as a part of snippet and maybe this is why it may rank pages that have the same text immediately visible higher.

However, it appears that it DOES impact rankings to a degree - but only if the exact sentence is copied by another site that has it immediately visible. The example of this is stackoverflow ranking at #1 when the text was searched for without quotes - although the originator of this text is Google.

Which is ultimately what Broadway said in his message above (and nicely sumarised in a much fewer words than me here!):

So, I think google does index the content, but doesn't give it the same elevated treatment as content that always shows on the page.