Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- SonicWALL Firewall Blocks Spiders


Hanu - 11:23 am on May 4, 2005 (gmt 0)


Ahh, another security frenzy!

Have a read of this...

a security site-based in Estonia, has uncovered the elementary mistake in RIAA's robots.txt files which gave the crackers their back door.

Don't stop there. Read on:

"This organization must be employing a blind webmaster if he did not figure out that this very passwordless admin module at www.thatsite.org/admin was used to deface the website. There was also no filtering to prevent uploading mp3 files through the PDF upload section. That would also explain how illegal mp3 music files appeared on this anti-piracy site," explained Holmes smugly.

Blocking any UA (spider or browser) that accesses robots.txt is surely not a solution. RTFM? Yes, but you should be able to assume that a high-end fireWall comes with reasonable default settings. Blocking anything that requests robots.txt is not reasonable. It's like cutting your phone line because you might receive prank calls.

Always remember the golden rule of IT security: Nothing is 100% secure. A secure system is one that has the right balance between prey value, attack effort, counter measure effort and usability impediments.


Thread source:: http://www.webmasterworld.com/robots_txt/631.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com