Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
The robots.txt file has been unchanged for weeks, definitely enough time for any willing spider to read it. Any ideas why it's still being ignored by this particular bot?
I'm still a little bit confused, though. According to that information, Google CHTML Proxy shouldn't be sniffing around in hidden directories that are only accessible via hidden links. (If it's just following human clicks, why does it follow a link that cannot be seen by humans?) Hm...
And I'm just going to suppose here, so: Suppose what's displayed in a chtml-capable browser is different than a "normal" html browser. Might it be that those "hidden" links aren't so hidden.