Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo! Slurp


g1smd - 6:01 pm on Sep 11, 2011 (gmt 0)


I found that the robots.txt in that one site had somehow become scrambled, such that all rules were on one continuous line
Are you sure they were all on one line? Perhaps the file format was using Unix-style LF line endings and you were using a text editor that required DOS/Windows-style CR/LF line endings to view it?


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4360952.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com