Forum Moderators: open
I'm a student writing a thesis about web usage mining. I'm interested in comparing different techniques for robot identification such as robots.txt, HEAD methods etc. I see there are a lot of people here who know a big deal about this. Could anyone tell me whether there is an up to date list with known robots? (i've already got the list from robotstxt.org, which is kind of useless..) Any other suggestions on ways to discover robots in a logfile are welcome!
Thanks
You'll find some good resources if you check the Library [webmasterworld.com] of this forum.
