Progressive Web App instead of responsive for mobile first index?

[nikhilrajr]

The smartphone bot crawl is more than 60% of the overall crawling on my website. A little background:

1) We are an ecommerce company.
2) the mobile website uses the same URL as desktop
3) We use dynamic serving; for any user-agent that can be identified as mobile or tablet we will show the Progressive web app site (PWA).
4) There is a big difference in the content present on desktop and PWA sites.
5) On the PWA there is no <a href> links even after the full rendering. It uses onclick action and takes users to the requested content. Not sure how Google smartphone is able to discover links though via the PWA. My guess is it's requesting the known desktop URLs via the smartphone bot.

So, we would have to create our PWA from scratch to support the Mobile first indexation in terms of content, design, crawlable hyperlinks etc.

I think below are my options for the moble first index:

1) Build the PWA from scratch - may take like 6 months or longer.
2) Make the desktop website responsive - Easy to do. But we have spend a lot of time and money on the PWA.

Now, this is what I have in mind. Please let me know your comments on this approach.

Make the desktop website responsive. Send all smartphone bot requests to the responsive website. Users will never see the responsive website, they will always see the PWA. I understand this is not the best option and goes against what Google is trying with their mobile first index.

But let me know what you guys think. Thanks!

[keyplyr]

Hi nikhilrajr, interesting topic and one that's becoming relevant quickly.

Send all smartphone bot requests to the responsive website.

I don't agree with that tactic. I would let Googlebot make that distinction on its own. I would not interfere with how Google discovers any links.

Surely the mobile-first indexing will accomodate PWA indexing. The responsive design just needs to point to the PWA. In affect, that *will* be the mobile version.

[NickMNS]

There is a third option that exists. That is to prerender the site server side and serve the pre-rendered version to Googlebot and others (bing,yahoo, etc...). The method is described and recommend by Google in this video from Google I/O 2018. It explains the best way to structure URL's and get your content crawled by Google when using Javascript.
[youtube.com...]

[Robert Charlton]

NickMNS, can you find a URL for that particular video. Google published a lot of video at I/O 2018, and I couldn't quickly find the one you were suggesting.

[NickMNS]

Sorry! I copied the link with some text so it didn't keep the hyperlink.
Here it is:
[youtube.com...]

[nikhilrajr]

I can tell after lots of debugging and talks with the DEV, the smartphone bot is unable to follow links via the PWA website. The PWA is not crawlable.

But, we are seeing more than 60% daily crawls from smartphone bot on the PWA which is surprising. So my guess is it's crawling the known Desktop URLs using the smartphone bot.

PWA rework for metadata, content, schema, canonical tag etc to follow the same as the desktop website will take at least 6 months of work. That's why I thought of detecting user agent and redirect only smartphone bot to the responsive website. I got this idea that this is ok after watching the Google IO event video regarding js websites.

Could you guys give me some example websites running on PWA especially from e-commerce industry. Also any resources on PWA and SEO. I found this from Google [developers.google.com...]

What I have learnt from my research is most of the competitors have moved to AMP.

And on the prerender - I don't agree this is the best solution. Our desktop website and PWA relies heavily on Javascript and is client side rendered. What we are moving towards is server-driven rendering frameworks on both desktop and PWA sites. These are pretty good when it comes to SEO and performance. There is something called hybrid rendering. For first page load requests the pages will load from server side, and for subsequent requests client side rendering will be used.

[keyplyr]

Could you guys give me some example websites running on PWA especially from e-commerce industry

If you just do a web search for "pwa examples" you'll get some.

[NickMNS]

And on the prerender - I don't agree this is the best solution.

I agree I don't think it is the best solution either. I would hate to have to build a complete pre-rendering system to run in parallel simply to feed pages to the bots. But if Googlebot and other bots can't see or follow your links I don't see how you have any other choice.

There is something called hybrid rendering.

Yes, sure. This is the approach that I take. But if your links are not rendered server side for the bots then your hybrid functionality is not sufficient.

One word about PWA's. The issue described has very little to do with the fact that a site has PWA functionality. This issue has only to do with the crawlability of the site given the use of Javascript and more specifically where, when and how the elements of the page are rendered. A PWA is simply added functionality that occurs in the background.

How is your website handling iOS devices that do not support service workers?

[nikhilrajr]

Yes, NickMNS. We are yet to start working on the hybrid rendering for the PWA. That's why I thought of showing the responsive website for the smartphone bot as an interim solution. iOS devices do support service workers, I think from early this year. You can check that here [caniuse.com...]

Also, I was trying to identify which all of our competition is in PWA and this was highly useful [quora.com...] to identify if they are running on PWA.

[NickMNS]

Clearly I missed that announcement. Busy with other non-pwa projects at the time. But after some digging it seems to still be somewhat limited, only available on iOS 11.3 and up and only in Safari (not in Chrome for iOS). I checked my stats iOS 11.3 and up accounted for more the 30% of my traffic in the last week (maybe not so limited...)

I found this article that provides a good summary and some of the differences:
[medium.com...]

[docluv]

nikhilrajr - I think you have confused a progressive web app with a single page app. A progressive web app is a web site served via HTTPS, has a valid web manifest file that references a minimal set of icons and has a registered service worker.

While a single page app can be a PWA, my personal position is PWA kills the need for a SPA because caching is now native and does not need to be polyfiled.

I don't post here much, but I used to be a specialist in SPAs, even wrote a book on how to create them and presented at many conferences, etc. The day I saw service workers demonstrated I knew SPAs were dead. Since then I have specialized in PWAs. Just had a new book published and have a course of over 21 hours, etc.

PWAs are excellent for SEO. But like any site is easy to screw up. Today far too many rely on fast food frameworks like Angular, React that are just killer JavaScript payloads. The Wordpress themes I see are loaded with excessive JS and CSS that keeps sites from just rendering. I think this is most likely the cause of your issues.

I recommend having a platform that pre-renders your HTML as static pages and is served via a CDN for better performance. If there are some areas that need to be dynamically updated on the client you can, but keep that to a minimum.

[NickMNS]

PWAs are excellent for SEO. But like any site is easy to screw up.

Do you care to elaborate. So far my experience working with PWA's is that SEO is more difficult as is monetization. There is also very little discussion about the topic here, so I would really value your view point.

[nikhilrajr]

Found these case studies and some other links which will be highly useful:

[developers.google.com...]
[divante.co...]
[medium.com...]
[paul.kinlan.me...]


Ok, so like I said the whole point is our PWA is completely JS, and Google won't be able to crawl the links within the PWA. But we are seeing more than 60% of daily crawls from Google smartphone bot.

Using log files, following is the crawl distribution for the PLA ad URLs - smartphone bot 45%, desktop bot 45%, admobile 2%, adsbot 3%. I have classified bots according to the Google user agent list mentioned here [support.google.com...] . Using Botify I have confirmed the numbers are true.

So, what's happening here? Actually, almost all the smartphone bot crawls are for the PLA ad URLs. Do you guys have any info on this? Which bot is supposed to crawl PWA ads? I thought about blocking all but allow only the admobile and adsbot for all Adwords URLs. But haven't done that as the data suggests PLA ads are crawled by desktop and smartphone bots than the ad bots.

[docluv]

I have heard the web is difficult to monetize for years. One word, Amazon....

Again a PWA is just a web site that takes advantage of modern technologies to deliver great user experiences.

I think you are still confusing a single page app with progressive web app. A SPA can be a PWA, but being a SPA does not mean you are a PWA.

Google can execute some JavaScript, but it is not 100% accurate for many technical reasons. I wont bother with that, too technical and does not really matter here.

The real problem is the JavaScript SLOWS the page down significantly and no one likes that. Today most web pages are loading over 1MB of JavaScript and 99% is never used, but this is the #1 reason why pages render slowly. And that is BAD UX and Google rewards good UX.

So to the point of monetizing a web site:

Can you sell products and charge credit cards - YES
Can you run ads that generate revenue or CTA - YES
Can visitors become subscribers that pay with their credit card - YES
Can visitors make a phone call to you by clicking a link on your phone number - YES
Can you integrate a chat bot to drive a visitor down your funnel - YES
Can you send push notifications to a subscriber - YES
Can you make content only available to subscribers - YES
Do You have to pay Apple, Google and Microsoft 30% per transaction plus annual app store fees - NO
Do Customers have to go from your site through an 8 step process to download your app to get to the same content that is on your web site - NO
Does a visitor/customer have to add your PWA to their homescreen - NO They still get all the benefits of your great UX minus the home screen icon and the ability to launch in full screen mode.

So I would like to know how you can't monetize your web site?

[Leosghost]

You should post more often ( if time and life permit )..it improves the S/N ratio here*..:)

Not referring to this thread as N..it is actually one of the more interesting ones.
I'd quote your post #4915975 in it's entirety , but easier to suggest that people just read it again, and again, and again, until it sinks in..

[NickMNS]

So I would like to know how you can't monetize your web site?

Maybe I should have been more specific. First let me say that I know the difference between an SPA and a PWA. I should also add that PWA or more broadly "service workers" provide a set functionalities, including off-line functionality, push notification and others. So not every website will use all the functionalities and they all wont apply them in the same fashion.

In my case, I have an informational website and I use the PWA to "power" a calculator app, specifically to allow it to work off-line. I do not offer any push notifications or anything of that nature. It is quite simple and straight forward. My issue with monetization is mostly around off-line functionality. I have simply been using AdSense ads, but obviously when off-line no ads display. This is problematic. So my question is really about the options for showing off-line ads and how are off-line clicks (if any) are tracked and handled within those solutions.