homepage Welcome to WebmasterWorld Guest from 54.166.159.110
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
ClamAV 0.95.3
GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4033694 posted 10:07 pm on Nov 29, 2009 (gmt 0)

ClamAV 0.95.3
212.40.106.nnn
morci.graphart.hu
-----
I hate when this thing comes calling. Darn thing took close to 2,800 files from one of my main money sites before I finally realized what was going on and stopped it.

 

jabz

5+ Year Member



 
Msg#: 4033694 posted 12:53 pm on Dec 3, 2009 (gmt 0)

Question is, does it obey robots.txt rules? Guess not, huh?

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4033694 posted 2:11 pm on Dec 3, 2009 (gmt 0)

My apologies, I forgot my usual robots snippet.

READ ROBOTS.TXT? No
OBEYED ROBOTS.TXT? No

jabz

5+ Year Member



 
Msg#: 4033694 posted 5:56 pm on Dec 3, 2009 (gmt 0)

afaik ClamAV (virus scan for linux) does not offer to check websites for viruses. Seems to be a name-highjack. Sad,...ClamAV is great.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4033694 posted 10:30 pm on Dec 3, 2009 (gmt 0)

Clamscan's command line includes a switch "mail-follow-urls" to "Download and scan URLs". I suspect that's it. As far as I know it's not enabled by default, although it may augment scans for phishing URLs.

It would need someone a bit clued up to set this as clam is poor on user interfaces.

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4033694 posted 4:09 am on Dec 4, 2009 (gmt 0)

Gary, you'd be spared a whole lot of bad, baaad hits -- from real and faked UAs -- if you whitelisted instead of blacklisted;)

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4033694 posted 2:54 pm on Dec 4, 2009 (gmt 0)

Because of my browser project I need to see how UAs behave so I usually let everything in and then only ban if the behavior is excessively egregious, like the Bing bots. It's because one of the things I do is recommend which UAs are ban-worthy. This version of Clam is now banned. But if a new one comes calling it'll be able to crawl until it does something bad enough to warrant my intervention.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved