Page is a not externally linkable
starchild - 11:14 am on Aug 1, 2011 (gmt 0)
Hi,
I saw today that Googlebot got caught in a spider trap that it shouldn't have as that dir is blocked via robots.txt
I know of at least one other person recently who this has also happened to.
Why is GB ignoring robots?