WebFilter Robot 1.0

Forum Moderators: open

Message Too Old, No Replies

WebFilter Robot 1.0

Good/Bad or just plain Ugly?

pendanticist

2:28 pm on Apr 3, 2003 (gmt 0)

63.173.114.243 - - [03/Apr/2003:05:52:39 -0800] "GET /robots.txt HTTP/1.1" 200 188 "-" "WebFilter Robot 1.0"
63.173.114.243 - - [03/Apr/2003:05:52:39 -0800] "GET /gifs/ HTTP/1.1" 200 14182 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)"

Site Search didn't provide anything.

No robots.txt violations.

Anyone have any experience?

Pendanticist.

wilderness

6:32 pm on Apr 3, 2003 (gmt 0)

Visited my sites on 2/16 and 2/22 viewing root and main folders. I added them to my denies.

http ://www.getnetspective.com/

WitchLars

10:52 pm on Apr 3, 2003 (gmt 0)

63.173.114.243 checked my robots.txt and main index page just before violating said robots.txt and getting itself promptly banned.

-Lars

carfac

7:08 pm on Apr 4, 2003 (gmt 0)

Strange:

>>>63.173.114.243 - - [03/Apr/2003:05:52:39 -0800] "GET /gifs/ HTTP/1.1" 200 14182 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)"

Why would a bot check /gifs?

Did it actuall GRAB images? Or just look through that directory for html?

dave

pendanticist

7:31 am on Apr 5, 2003 (gmt 0)

>Why would a bot check /gifs?
<shrug> Dunno.

>Did it actuall GRAB images? Or just look through that directory for html?
What I posted is all there was that visit.

Last night it was back...

63.173.114.243 - - [04/Apr/2003:20:06:42 -0800] "GET /robots.txt HTTP/1.1" 200 188 "-" "WebFilter Robot 1.0"
63.173.114.243 - - [04/Apr/2003:20:06:43 -0800] "GET / HTTP/1.1" 200 20363 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)"

See how after the first request it again switches away from WebFilter Robot 1.0? This time, however, it did not ask for "/gifs/", but rather a more standard "GET".

Still, these are the only requests...so far.

Pendanticist.

wilderness

2:10 pm on Apr 5, 2003 (gmt 0)

Pendanticist
When they visted me it was just pages. No images.

On a side note, there are a few bots that change their UA after reading robots. Lycos comes to mind first. They read robots with both blank referrer and UA and then change to their standard UA for pages.
Their is another active bot which which changes from UA to no UA. I'm just at a loss this AM :(

pendanticist

3:07 pm on Apr 5, 2003 (gmt 0)

>I'm just at a loss this AM :(

That's ok. When you remember it, let us know....

Pendanticist.

wilderness

4:59 pm on Apr 5, 2003 (gmt 0)

Only went through my largest March log looking at occurances after reading robots.

66.65.80.221 - - [01/Mar/2003:03:50:48 -0800] "GET /robots.txt HTTP/1.1" 403 - "-" "Java1.4.0_03"
66.65.80.221 - - [01/Mar/2003:03:50:49 -0800] "GET / HTTP/1.1" 403 - "-" "RPT-HTTPClient/0.3-3"

63.173.114.243 - - [31/Mar/2003:11:29:48 -0800] "GET /robots.txt HTTP/1.1" 403 - "-" "WebFilter Robot 1.0"
63.173.114.243 - - [31/Mar/2003:11:29:49 -0800] "GET /myfolder/ HTTP/1.1" 403 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)"