Welcome to WebmasterWorld Guest from 23.22.220.37

Message Too Old, No Replies

Blocked CSS & JS apparently causing large Panda 4 drops

     
12:01 am on Jun 20, 2014 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:11309
votes: 163


This interesting case study by Joost de Valk may explain some of the recent traffic drops reported in this forum. Joost dealt with several sites which reported losing roughly 2/3 of their traffic, at roughly the time of Panda 4....

Google Panda 4, and blocking your CSS & JS
19 June, 2014 by Joost de Valk
https://yoast.com/google-panda-robots-css-js/ [yoast.com]

He associated a site's drop with its accidental blocking of CSS and Javascript files. This was shortly after Google had announced its new Fetch and Render feature in Webmaster Tools. Assumption is that this is now being used in the page layout algorithm. Unblocking CSS and JS appeared to produce quick recoveries.

This kind of association is entirely consistent with algorithmic changes I've seen over the years, where Google has been quick to make use of a capability that we first see in a reporting feature.

See our discussion here...

Rendering pages with Fetch as Google
May 27, 2014
http://www.webmasterworld.com/google/4675119.htm [webmasterworld.com]

In this earlier Google blog post, Google strongly hints at the importance of not blocking JSS and CSS....

Understanding web pages better
Friday, May 23, 2014
http://googlewebmastercentral.blogspot.ca/2014/05/understanding-web-pages-better.html [googlewebmastercentral.blogspot.nl]

If resources like JavaScript or CSS in separate files are blocked (say, with robots.txt) so that Googlebot canít retrieve them, our indexing systems wonít be able to see your site like an average user. We recommend allowing Googlebot to retrieve JavaScript and CSS so that your content can be indexed better. This is especially important for mobile websites, where external resources like CSS and JavaScript help our algorithms understand that the pages are optimized for mobile.

Anyone seeing drops since Panda 4 which can be attributed to this change?
7:41 am on June 20, 2014 (gmt 0)

Junior Member

joined:Sept 12, 2012
posts: 89
votes: 1


I'm just doing some tests, unblocking all JS and CSS that render my pages.
Let's see.
8:04 am on June 20, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2557
votes: 48


I have read some related complaints in comments on the Yoast article that people who are using Google Web Fonts are seeing Google messages about the blocked css when the only blocking is on the Web Fonts site itself. I don't use their Web Fonts, but I would probably check into that if I did.
8:20 am on June 20, 2014 (gmt 0)

Full Member

5+ Year Member

joined:May 25, 2006
posts:217
votes: 7


This sounds both sensible and easy to do.

But before I rush in and remove 'templates' from my robots.txt which would enable googlebot to access the css files and render the page better, can anyone see any downside or risks to doing this or are there only potential benefits?

For example, could a site that is not affected by 'above the fold' penalties or is not well designed for mobiles suddenly find it is exposing its shortcomings which were previously not visible? (I don't think these two apply to me, but wonder if there are other risks)
8:51 am on June 20, 2014 (gmt 0)

Preferred Member

Top Contributors Of The Month

joined:Mar 12, 2013
posts:500
votes: 0


What reasons would a webmaster have to block such files from Google?
11:00 am on June 20, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12671
votes: 141


Some people block everything that they don't think Google has a good reason to see.
11:03 am on June 20, 2014 (gmt 0)

Preferred Member

Top Contributors Of The Month

joined:Mar 12, 2013
posts:500
votes: 0


Personally, I don't know what the problem would be for Google to see the CSS file I'm using.
12:01 pm on June 20, 2014 (gmt 0)

Moderator This Forum from GB 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Apr 30, 2008
posts:2507
votes: 138


What reasons would a webmaster have to block such files from Google?

I am blocking .js files. The reason is that certain links create URL with date parameter. This creates many URLs that are the same, the only difference being the date parameter in URL.

I know that I could use parameter handling in WMT or robots.txt to block such URLs, but it seemed a cleaner solution not to even let Google know such URLs exist.

In fact they are also blocked by robots in case Google pick up some of them via different means (e.g. external link), but blocking these URLs by creating the link in .js which is blocked from Google does not expose them to Google in the first place. Otherwise, every time Google crawled the page, it may see a different URL.

I may decide to unblock it on one of sites and see if there are any impacts on traffic/ranking.
12:05 pm on June 20, 2014 (gmt 0)

Junior Member

Top Contributors Of The Month

joined:Mar 24, 2014
posts:119
votes: 6


I don't think Javascript blocking is a problem per se. Adsense is blocking the Google bot by default, as do many other ad networks.

Blocking JS and CSS related to the style of a website might be a problem though, for the reasons told.

I've not seen any negative impact so far (knock on wood) for the blocked ad networks (adsense and another one) javascript files. I think Google can differentiate those from other scripts.
1:26 am on June 21, 2014 (gmt 0)

Preferred Member

Top Contributors Of The Month

joined:May 24, 2012
posts:648
votes: 2


The irony is that many people probably blocked access to avoid having Panda count stylesheets and scripts as "thin content".

I wonder if marking the context as "noindex" with an X-Robots-Tag in the http header has the same adverse affect. It's not blocking it per se, but it wouldn't be in the cache, if that's where Panda was expecting it to be.
2:18 am on June 21, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12696
votes: 244


to avoid having Panda count stylesheets and scripts as "thin content".

Oh, come on. Google's mind may not work like yours and mine, but they know a stylesheet when they see one.
3:58 am on June 21, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 6, 2002
posts:1825
votes: 21


They spy on everything, lol do you really think you can block spying with robots.txt come on

They even know what you eat, so forget about robots.txt
6:24 am on June 21, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Mar 9, 2010
posts:1806
votes: 9


funny...I am actually seeing an increase in traffic after Panda 4 and everything within wp-content folder is blocked by robots.txt. I don't see any relationship between panda 4 and the css/js blocking.
6:28 am on June 21, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12696
votes: 244


do you really think you can block spying with robots.txt

As far as anyone knows, the Googlebot's visits are plainly visible in your site logs. Now, if you wanted to venture into tinfoil-helmet territory, you could postulate that they're paying off the major hosts to have certain things omitted from logs. Heh, heh. But this kinda falls apart if it's your own server.
9:49 am on June 21, 2014 (gmt 0)

Full Member

5+ Year Member

joined:May 25, 2006
posts:217
votes: 7


@indyank, I'm in a similar position with joomla which also blocks the relevant directories by default.

The link with panda 4 seems tenuous but it's not impossible this css/js blocking could explain other smaller changes that take place from time to time for example in mobile search. Have you considered removing the directories from robots.txt anyway to see if, for example, the mobile version of your site gets better visitor numbers?
11:36 am on June 21, 2014 (gmt 0)

Preferred Member

Top Contributors Of The Month

joined:Mar 12, 2013
posts:500
votes: 0


Why would it be a Panda-related issue if it's more to do with layout? If they can't load the css file(s), they can't determine if your site is "top heavy" with ads, or the text colour of the block of text at the bottom of your page (big difference if it's white on white background, or black on white background!) - they want to see the page as a human sees it (i.e. they need to load the css to render the page as a human would see it).
11:42 am on June 21, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts:3224
votes: 9


They have removed the yoast article, but it is still in the cache. I wonder why it was removed.
12:12 pm on June 21, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member lame_wolf is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Dec 30, 2006
posts:3224
votes: 9


I've just tried using fetch and render on Webmaster Tools. It gave the URL a "partical" pass.

It says that fonts.google and two of Google's Adsense .js files were blocked by robots.txt

Whose robots.txt blocked it? It wasn't mine, as I do not have anything blocked in my robots.txt file.

Any ideas? and if is something to be concerned about?
Thank you
1:07 pm on June 21, 2014 (gmt 0)

Preferred Member

Top Contributors Of The Month

joined:May 24, 2012
posts:648
votes: 2


Oh, come on. Google's mind may not work like yours and mine, but they know a stylesheet when they see one.


They know a tag page when they see one as well, yet there's 1001+ wordpress plugins to noindex it so Panda won't count it as duplicate content.
4:16 pm on June 21, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:June 28, 2013
posts:2367
votes: 208


to avoid having Panda count stylesheets and scripts as "thin content".


You might want to delete your robots.txt file, too, if it's only a few lines long. :-)
7:07 pm on June 21, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12696
votes: 244


It says that fonts.google and two of Google's Adsense .js files were blocked by robots.txt

Whose robots.txt blocked it?

Didn't this come up in an earlier thread? Honestly, google, how hard would it be to give a pass to your own robot in your own robots.txt? :)

Don't all the standard cms "themes" come with standard named css? So even if a search engine can't crawl them, it pretty well knows what's there. That's assuming for the sake of discussion that there aren't vast numbers of sites pretending to use a standard CMS by making the html look just like WordPress or Joomla, and intentionally giving their personal stylesheets conventional names, just to pull the wool over g###'s eyes.

You might want to delete your robots.txt file, too, if it's only a few lines long.

<FilesMatch "\.(js|txt|xml)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

Not because of thin content but because I don't trust them not to index it otherwise. The only reason css is exempt is that it's in a different envelope.
7:56 pm on June 21, 2014 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:11309
votes: 163


Getting back to the topic of the Yoast article, it helps to read the article and to note the particular types of pages this problems appears to have affected.

As Joost describes the page he used as an example....

Now, iPhoned makes money from ads. It doesn't have a ridiculous amount of them, but because it uses an ad network a fair amount of scripts and pixels get loaded. My hypothesis was: if Google is unable to render the CSS and JS, it can't determine where the ads on your page are. In iPhoned's case, it couldn't render the CSS and JS because they were accidentally blocked in their robots.txt after a server migration.

As I check the page linked to from the article, the page doesn't look like it should trigger an above-the-fold ad algo; but looking at the (only partial) screen capture of how Fetch as Googlebot saw the page... and also looking at the page source code... I can see how blocking CSS and JS in this case could well cause Google to misinterpret the page.

The article also quotes Maile Ohye's recent comments regarding how Google uses the new Fetch and render capability....

We recommend making sure Googlebot can access any embedded resource that meaningfully contributes to your site's visible content or its layout

If this is part of Panda 4... and we're not sure of that but it does come nearby in time... how does this mesh with the "kindler, gentler" aspects proclaimed for the algo? I'm guessing that it may be kindler and gentler because it's correcting for an ad layout that wasn't visibly distracting and shouldn't have been hit... but it can only do this if it "sees" the layout, and for that the bot needs to be able to see CSS and JS.

IMO, Google's "rendering" is there to make a finer discrimination, not to screw you. YMMV.

Does this mean that all JS should be open to Googlebot? In my book, no. But if you're in this kind of situation where you've been hit by a large drop and have a layout that might look better to an above-the-fold algo if the page were rendered properly, you might want to unblock the code affecting potentially applicable areas.
8:14 pm on June 21, 2014 (gmt 0)

Full Member

joined:Feb 25, 2011
posts: 255
votes: 0


I block my CSS and JS, I got 150% rise from panda 4. Its more likely page layout algo which im sure runs constantly, so a good crawl and away you go.
However, my page renders similar and its booming at the minute so ive no intention of changing.

As for why block it, originally it was to block google know what plugins I used for SEO..Google cause these issue by keeping us in the unknown.
9:25 pm on June 21, 2014 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 2, 2004
posts:125
votes: 0


We lost 30% of our traffic on panda 4 across multiple language sites. All sites have very similar layout. Approximately 20% of the above the fold content is taken up by a google maps image using their js plugin. I tried running an example webpage through the "fetch and render" and the map is completely missing and there is just the background colour without any content or image. The google maps addin apparently blocks googlebot on its robot.txt. Not sure if people have had similar issues with google maps.
9:34 pm on June 21, 2014 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 2, 2004
posts:125
votes: 0


And the map not showing looks even worse on the mobile version of our site.
9:54 pm on June 21, 2014 (gmt 0)

Preferred Member

Top Contributors Of The Month

joined:Mar 12, 2013
posts:500
votes: 0


I say again - why are people confusing css with content? Panda is about content, no? So why are we saying if you block .css files, you run the risk of a Google content filter (Panda) punishing you? It's far more likely to be a layout related issue .... because css.... is layout....! And if you deny Google the chance to see not only your actual layout, but text colours and sizes, then of course Google aren't going to trust your site as much. How can they know your text colour for a block of text on a page isn't white on white if they can't render the css?
10:11 pm on June 21, 2014 (gmt 0)

Senior Member from AU 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 22, 2003
posts: 1942
votes: 91


Hmmm. I just did a GWT "Fetch and render" of the home page, and received this message:

URL/Type/Reason

1. http://translate.google.com/translate_a/element.js?cb=googleTranslateElementInit - Script Denied by robots.txt

2. http://pagead2.googlesyndication.com/pagead/show_ads.js - Script Denied by robots.txt [Google AdSense]

3. http://edge.quantserve.com/quant.js - Script Denied by robots.txt

4. http://kona.kontera.com/javascript/lib/KonaLibInline.js - Script Denied by robots.txt

5. http://www.example.com/logo2.jpg - Image Denied by robots.txt

6. http://www.example.com/ArrowOrange.gif - Image Denied by robots.txt

So, how does one allow Googlebot access? Why is this suddenly a problem, particularly 1 & 2? All the others were image files.

Is this really a problem? GWT otherwise reports no fetch problems.

[edited by: aakk9999 at 10:42 pm (utc) on Jun 21, 2014]
[edit reason] Unlinked sample URLs [/edit]

11:50 pm on June 21, 2014 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 2, 2004
posts:125
votes: 0


The issue is Googlebot cant see the content most humans see on the web. If this fetch and render thing forms the basis of panda 4 then its clearly quite flawed.
12:45 am on June 22, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2557
votes: 48


@IanCP, you are seeing exactly what others have complained about - it isn't your robots.txt that is blocking access, it is their own.
12:57 am on June 22, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:6134
votes: 279


Are we really sure what Panda 4 is? Reason why I ask is that a few sites I manage (for others) which do not use either CSS or JS saw a drop on the roll out of Panda 4...

Inquiring minds want to know. Rather than speculate and chase unsubstantiated ghosties.
This 43 message thread spans 2 pages: 43