danwhitehouse - 9:37 am on Mar 27, 2012 (gmt 0)

Hi all,
We've just changed URL manager, which has left a great number of redundant URLs live. A lot of them seem to have the same convention, an '_' in them.

I've thought about creating a script that detects urls with '_' in them and adding them to Robotx.txt on the fly with a cron running in the background.

I was wondering if anyone knew of a faster way to do this?

Thanks again