stinkysloppy - 12:36 am on Jul 19, 2008 (gmt 0)

Apparently I have had hundreds of GB of traffic going to yahoo slurp on my site

I needed this to stop IMMEDIATELY....

I believe it is because slurp for some reason is downloading my FLV files - I have
no clue why it would need to do this but whatever.....

So I am testing some new code in my robots.txt file that hopefully should
eliminate this problem....

user-agent: *
Disallow: /flvplayer.swf

User-agent: Googlebot
Disallow: /*.flv$

User-Agent: Yahoo! Slurp
Disallow: /*.flv$

So if you have a website with various media files, you may be able to tell the bot
to not download them.

Here are the resources I used to build these directives.

[google.com...]

[ysearchblog.com...]