- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- Yahoo now violating robots.txt
volatilegx - 6:52 pm on Dec 28, 2007 (gmt 0)
|but always only asking for the html pages, no supporting files - as it should if respecting our robots.txt. |
What do you mean? You can't require them to ask for files via robots.txt.
Thread source: http://www.webmasterworld.com/search_engine_spiders/3536187.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com