Making Infinite Scroll Search Crawler Friendly

[engine]

Your site’s news feed or pinboard might use infinite scroll—much to your users’ delight! When it comes to delighting Googlebot, however, that can be another story. With infinite scroll, crawlers cannot always emulate manual user behavior--like scrolling or clicking a button to load more items--so they don't always access all individual items in the feed or gallery. If crawlers can’t access your content, it’s unlikely to surface in search results.

To make sure that search engines can crawl individual items linked from an infinite scroll page, make sure that you or your content management system produces a paginated series (component pages) to go along with your infinite scroll. Making Infinite Scroll Search Crawler Friendly [googlewebmastercentral.blogspot.com]

You think infinite scroll is cool? Search-friendly infinite scroll is even cooler! Crawling the infinite scroll [googledevelopers.blogspot.com]

With some implementations of infinite scroll, crawlers aren’t great at scrolling down or clicking “load more”, so they may not crawl items displayed after the initial page load. To help the crawler see all the content, we recommend converting the infinite scroll page to paginated series by using the HTML5 History API. (Of course, the pagination is seamless to the user.)

[goodroi]

In other words be careful when using the latest technology because Googlebot might have issues. Keep it as simple as possible to avoid Googlebot problems.

[graeme_p]

Now, what about making infinite scroll user friendly?

[goodroi]

User friendly is a very big issue because there are so many different types of users. Mobile vs desktop users, high-speed vs low speed, older users with vision problems, users speaking different languages, older browsers vs. newer browsers, etc.

When implementing infinite scroll into a website, I would say it is more user friendly to ensure there is an alternative to infinite scrolling. Once there is an alternative for each major segment of your users, you can then figure out how best to address each segment.

If you have many users on slower connections you probably want to be more careful on how much additional information you load when they scroll down.

If you have older visitors with poor vision, then you probably want to display larger thumbnails if your infinite scroll is a photo gallery.

If you have a massive archive, then you might want to add shortcuts so a user does not have to scroll down for 10 minutes to see your April 2012 content.

Infinite scrolling can be a good thing for users if the webmaster thinks through the many different issues that it can possibly introduce.

[jskrewson]

So, my advice from a usability perspective is to **REALLY** think about how your users use your website before you implement infinite scroll.

I recently spent some time on a B2B site that thought it would be a good idea to implement infinite scroll in their search results. What they didn't consider was that their search results can be quite long and their users might want to spend DAYS scanning the results, in detail.

Infinite scroll, in conjunction with long results lists and extended browsing sessions is a recipe for an extremely frustrated and angry user base. If result sets are fairly static (e.g. sorted alphabetically), then the user might want to the ability to quickly jump back to a previous position. (say the 'W') section).

In my opinion, infinite scroll is cool for mobile devices, but, just because it works well with handheld computers, doesn't mean its appropriate for all audiences. Don't make the Windows 8 mistake!

[incrediBILL]

In other words be careful when using the latest technology because Googlebot might have issues. Keep it as simple as possible to avoid Googlebot problems.

That's one theory while Google is promoting Angular JS which is barely crawlable at all!

Some of their demos show a page full of empty <div> tags that get replaced with pages and content on the fly, much like infinite scroll, so if Google can promote such technologies you know they're figuring out how to index them as well.

Most importantly, experiment first before leaving you or a big customer out to dry. Find out what Google's limitations are before investing heavily in tech that the SE's can't crawl. Also, make sure they can ALL crawl your site because if Bing and others can't crawl the site and you get a penalty in Google then >POOF!< you are completely gone.

Also try investigating alternative methods of exposing the same data to search engines.