| Ask Jeeves robots.txt entry
what UA should be used |
Kerrin
msg:1525375 | 4:10 pm on Nov 7, 2002 (gmt 0) |
Mozilla/2.0 (compatible; Ask Jeeves) Hi Guys, I'm a little confused as to what I should call the Ask Jeeves/teoma/directhit crawler in my robots.txt file. What should I use?
|
jdMorgan
msg:1525376 | 4:19 pm on Nov 7, 2002 (gmt 0) |
Kerrin, User-agent: Ask Jeeves Jim
|
Kerrin
msg:1525377 | 4:40 pm on Nov 7, 2002 (gmt 0) |
Thanks Jim, I've seen robots.txt including teoma, directhit and a few varients as UA's for aj.com's crawlers, I just wasn't sure what UA Ask Jeeves actually looks for in robots.txt.
|
|
|
