Forum Moderators: DixonJones
This one also comes at me from both IP Numbers within the same block.
209.133.XXX.21 - - [25/Aug/2004:18:31:36 -0700] "GET /robots.txt HTTP/1.0" 200 1681 "-" CosmixCrawler/0.1"
209.133.XXX.28 - - [25/Aug/2004:18:32:02 -0700] "GET /Veterans.html HTTP/1.0" 200 11269 "-" "falcon/1.0"
[webmasterworld.com...]
In this thread I also noted Lycos_Spider_(modspider) has had different UAs as well.
Is there some methodology I should / could be employing?
Thanks.
[edited by: webdiversity at 7:35 am (utc) on Sep. 1, 2004]
[edit reason] No specifics please [/edit]