Iac

[wilderness]

Jeeves, Ask and their numerous crawls have been a pest topic in many, many discussions.

And now IAC adds more UA's, and significantly, terminology that assures denial.
Go figure!

66.235.112.z - - [19/Jan/2009:12:59:32 +0000] "HEAD / HTTP/1.1" 403 - "-" "DeadLinkCheck/0.4.0 libwww-perl/5.803"
66.235.112.z - - [19/Jan/2009:12:59:33 +0000] "GET / HTTP/1.1" 403 1012 "-" "DeadLinkCheck/0.4.0 libwww-perl/5.803"
66.235.112.z - - [19/Jan/2009:13:02:02 +0000] "HEAD /MyFolder/MyPage.html HTTP/1.1" 403 - "-" "DeadLinkCheck/0.4.0 libwww-perl/5.803"
66.235.112.z - - [19/Jan/2009:13:02:02 +0000] "GET /SameFolder/SamePage.html HTTP/1.1" 403 998 "-" "DeadLinkCheck/0.4.0 libwww-perl/5.803"
66.235.124.zzz - - [19/Jan/2009:13:32:34 +0000] "GET /robots.txt HTTP/1.1" 200 5023 "-" "-"
66.235.124.zzz - - [19/Jan/2009:13:32:34 +0000] "GET /DifferentFolder/DifferentPage.html HTTP/1.1" 403 998 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20070308 Minefield/3.0a1"

[caribguy]

---

System: The following message was spliced on to this thread from: http://www.webmasterworld.com/search_engine_spiders/3838699.htm [webmasterworld.com] by incredibill - 5:46 pm on Jan. 30, 2009 (PST -8)

---

Following up on two posts from early 2008:

[webmasterworld.com...]
[webmasterworld.com...]

I'm seeing requests for robots.txt without a user agent, followed by crawls from the same ip address with this UA:

"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a1) Gecko/20070308 Minefield/3.0a1"

66.235.124.aaa
78.137.163.bbb

Where I also note that

66.235.124.ccc (multiple ip's) use

"Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)"

Two questions:

1. Dare I ask our esteemed moderator what makes him believe 78.137.163.bbb to be anything other than a host for the Dublin-based European division of Ask/Teoma? Different behavior maybe?

See also [sp.uk.ask.com...]

2. Not being a believer in either spoofed UA's or thumbnail generators, I'd like to confine 66.235.124.aaa as well. Good or bad?

Thanks!