Forum Moderators: goodroi

Message Too Old, No Replies

Getting correct user agent

getting from access log the correct user agent for robots.txt

         

hessodreamy

10:08 am on Jan 9, 2008 (gmt 0)

10+ Year Member



So in the access log there's user agents like this:
"TwengaBot/1.1 (+http://www.twenga.com/bot.html)"
"Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

How do I get from this the correct user agent to put in the robots.txt? ie Ask Jeeves/Teoma - is this the correct user agent or just one of them?

Also is the user agent case sensitive? Does it matter if i user twengabot instead of TwengaBot?

goodroi

9:03 pm on Jan 11, 2008 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Robots.txt is a voluntary protocol. That means even if you list a bot in your robots.txt file it can decide to ignore your robots.txt file. The major engines try their best to obey robots.txt.

Here is a good list of the major bots [robotstxt.org...]