Do Search Engines Spider CSS?

What happens if you block search engines from spidering your CSS file?

[DigiFundi]

Hi,

I have a couple of questions on the effects of using CSS on SEO.

Do search engines spider CSS files? So for example say if you have made a piece of text invisible using CSS would the spider know that you have done this? If they do spider CSS files - what happens if by using the robots.txt file you forbid the spider from reading the CSS file? Will it obey your instruction? Will it penalise you for forbidding it from doing so? And if it is forbidden from doing so will it know what affects you have created with the CSS?

Do spiders go through the code view of the webpage or do they go through users view of the webpage - or both?

Thanks for your time.

DigiFundi

[ChadSEO]

I don't believe that spiders download the css file - if they do, they are very sneaky about it. I have a webserver that hosts a dozen or so domains, and have access to server logs for 2 others - one of which gets a large amount of traffic, 50k page views/day, and is spidered daily. A search engine bot has never downloaded an external css or javascript file from any of these domains - nor has a normal user agent coming from the same class-c subnet that is also used by a bot.

There are no blocks from downloading the css files for any of these sites - no robots.txt entries, no .htaccess weirdness.

[DigiFundi]

Thanks for responding. That is a good point.

Then why is it often said that certain CSS techniques should not be used? For example in accessibility it is common to have a "Skip to Content" link. Some designers will hide the link by either positioning it off screen, or by making it invisible, or by making the text colour the same as the background colour all via CSS.

People say that these techniques should be avoided as Search Engines could consider it as spamming technique. Is there any evidence to support these concerns? If Spiders don't go near CSS then how would they know that certain content is not viewable to the standard browser user?

DigiFundi

[ChadSEO]

Certain CSS can be read: styles in the header (<style></style>), and inline styles (style="") can certainly be read by the search engines. As for the concern from other people, I'm not sure why they are concerned, or if they are justified. That is a question probably best left up to someone far more knowledgeable than I.

[MichaelCrawford]

I think what would really matter is whether you really are spamming the search engines.

There was a famous incident a couple months back where some prominent open source developer was selling links off his high PR homepage, then hiding them with CSS. Once discovered, it was plainly obvious that he was accepting pay to steal from legitimate adsense publishers.

Googleguy posted a few hours later where it was originally reported that Google's staff was removing his site and everything he linked to to from Google's index. His whole site is PR 0 and I expect will stay that way, and I'm sure he's busted broke and discredited in the open source community.

Having a "skip to content" link wouldn't deserve that kind of swift, harsh punishment, now would it?

Go find the actual google guidelines for webmasters, and apply them conscientiously and I'm sure you'll do fine. Google's not going to think your gaming them because you hide a couple links. Google's going to think you'ge gaming them because you're hiding a couple thousand of them.

[DigiFundi]

Fair points made.

Thank you.

[Marcia]

As an added point of interest, they can download CSS files if they choose to. Alta Vista was reported to be grabbing CSS files several years ago, when they were still independently alive and crawling.

[DigiFundi]

Hi Marcia,

Thanks for pointing that out.

Regards,

DigiFundi