PeterD
msg:1528980 | 1:51 am on Aug 31, 2004 (gmt 0) |
A proxy doesn't have to respect robots.txt because it's not a spider. There's a person behind it. Check out [webmasterworld.com...] Pete
|
yosmc
msg:1528981 | 2:14 am on Aug 31, 2004 (gmt 0) |
Thanks Pete, still learning something new every day. :) I'm still a little bit confused, though. According to that information, Google CHTML Proxy shouldn't be sniffing around in hidden directories that are only accessible via hidden links. (If it's just following human clicks, why does it follow a link that cannot be seen by humans?) Hm...
|
jimbeetle
msg:1528982 | 2:35 am on Aug 31, 2004 (gmt 0) |
This WW thread might help: [webmasterworld.com...] And I'm just going to suppose here, so: Suppose what's displayed in a chtml-capable browser is different than a "normal" html browser. Might it be that those "hidden" links aren't so hidden. Just specualtion.
|
yosmc
msg:1528983 | 12:16 pm on Aug 31, 2004 (gmt 0) |
Ah ok, thanks - now it makes sense.
|
|
