Blocking CSS Files in Robots.txt Can Cause Problems

[engine]

John Mueller has a quick answer session for the question of blocking CSS files in robots.txt, along with other topics of new sites, and RSS crawling.


I'm not entirely sure about the speed of refresh for a new site as it depends on so many factors, and John hasn't really gone into that aspect.

[not2easy]

Way back, when they started their Fetch as Googlebot feature they started warning people about blocking resources, css, .js, etc. because they could not evaluate a page's performance without it. I guess this is new though for sites started after that because the old Fetch as Googlebot does not seem to be around anymore and their new replacement does not easily share the information. Good idea to explain it. It used to be easier to get them to notice an updated robots.txt file.

[phranque]

a crawler will only notice the resulting changes in the response to specific urls being requested.
a crawler should never see or "notice a robots.txt file update".

[not2easy]

Sorry, I wasn't very clear there. When they had the Fetch feature you could edit and test your robots.txt file in GSC against resources they showed as blocked and then upload a robots.txt file that had been updated and click for them to retrieve it. I did not mean the crawler would notice it, just that you had tools in GSC to get it noticed.

[phranque]

maybe i should start my days with coffee again!

please disregard my comments above - when i read your post regarding robots.txt, i was thinking about .htaccess...

[lucy24]

Whew. Yes, it’s a case of “for ‘never’ read ‘always’” * since a law-abiding crawler most emphatically should notice a robots.txt change :)

Then again, letting them crawl your css doesn’t mean you have to let them index it. You could globally attach a noindex header.

* I have been known to go running to the nearest moderator immediately after the edit-post cutoff, beseeching them to add or remove a crucial “not”.

[tangor]

Never bothered dealing with css in robots.txt ... after all, good bots will obey the directives and all the others will not.

I allow two, three useful bots, disallow all others, so pretty easy to see who is compliant and who is not.