css & robots

do robots index css

[claysmc]

Hi,

Do robots index css pages and should
those pages be listed in a robots.txt
file to disallow indexing of those pages?

Thanks

[troels nybo nielsen]

Welcome to WebmasterWorld, claysmc.

Just checked a large search engine for one of my css files, a file whose name has been read thousands of times by the robots in the last year or so.

It's not there. And indeed there would be no idea for a search engine in indexing my css files. Not much informational value in them.

[Krapulator]

I suspect that in the coming couple of years we will see an increase in robots reading css files. Not for the purpose of indexing them, but for the purpose of detecting possible misuse for spamming purposes.

[troels nybo nielsen]

I agree and I really should have mentioned that aspect in my first post.

[tombola]

And if you exclude robots from your css directory (in your robots.txt file), you'll get penalized because you don't allow robots?...

[troels nybo nielsen]

> And if you exclude robots from your css directory (in your robots.txt file), you'll get penalized because you don't allow robots?...

Most likely the consequences will be less dramatic: You will invite someone to come and handcheck your website and especially your css files.

[transactiongeek]

Yeah, I suspect denying access to the css file would be a bad idea. Google especially needs to make sure you're not fiddling with the colors to do odd cloaking things (eg, matching foreground/background colors)

[dnimrodx]

Hello,

Can anyone give me a general idea of how the

robots.txt

file works and, also, how the Bots index sites?

Thanks.

[troels nybo nielsen]

robots.txt forum [webmasterworld.com]