Slurp now fetching CSS

Is this a new behavior?

[jdMorgan]

I just noticed Slurp/3.0 from llf320065.crawl.yahoo.net fetching included CSS files after fetching the page that includes them.

The first time I remember seeing this behavior, it was with the "Bonecho" spoofed Firefox user-agent mentioned in this thread [webmasterworld.com]. However, now it's the Slurp/3.0 user-agent doing it -- and providing a referer:

67.195.37.180 - - [28/Oct/2008:04:42:02 -0400] "GET /robots.txt HTTP/1.0" 200 1910 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.37.180 - - [28/Oct/2008:04:42:03 -0400] "GET /page.html HTTP/1.0" 200 11562 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"
67.195.37.180 - - [28/Oct/2008:04:48:05 -0400] "GET /style.css HTTP/1.0" 200 1210 "http://www.example.com/page.html" "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)"

Note the several-minute delay between the page fetch and the CSS file fetch.

I would guess that anyone "hiding" content on their page using the CSS {display: none;} technique might want to re-think that approach... :)

Has anyone else seen this before? Do you think they're checking for CSS "content cloaking?"

Jim

[wilderness]

Hey Jim,
I recall seeing delayed requests for CSS from one of the search engines, however not sure which as I did not document the reference.

Don

[blend27]

--- "hiding" content on their page using the CSS {display: none;} technique might want to re-think that approach -

we already did, external CSS File(s) are generated on the fly :)

text-indent: <cfif not Slurp-999</cfif>9px;