Forum Moderators: open
65.214.44.195 - - [12/Feb/2006:06:26:56 -0800] "GET /robots.txt HTTP/1.0" 403 - "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)"
Don't these bots explore the internet before creating new UA names? (i.e. Lycos "Spider")
Why did you post in "Search Engine Spider Identification"?
It took me more than a day to realize my mistake ;)
After pouring over my htaccess looking for a syntax error.
Finally caught a line that I had missed initially.
On Feb 1st I added the UA crawl for the following visit:
71.39.139.xx - - [01/Feb/2006:16:56:23 -0800] "HEAD / HTTP/1.1" 200 0 "-"
"test crawler"
It's now been modified.