Robots.txt sabotage

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt sabotage

Common? How to protect?

adfree

11:00 am on Sep 30, 2004 (gmt 0)

I heard about one case of robots file sabotage from a competitor. How common is it and how do you protect yourself?

trillianjedi

11:02 am on Sep 30, 2004 (gmt 0)

Can you explain what you mean by "robots.txt sabotage"?

adfree

11:03 am on Sep 30, 2004 (gmt 0)

Someone became aware that the robots.txt file was altered disallowing everything for all bots.

trillianjedi

11:05 am on Sep 30, 2004 (gmt 0)

OK.

Protecting yourself comes down to securing your webserver.

The robots.txt file is just a file on your webserver - if security on your webserver is compromised, that file can be altered just as easily as any page.

Change passwords etc. If you feel it might be an inside job, I'd run a script keeping an eye on the date/time stamp of the robots.txt file (that in itself could be easily compromised though).