homepage Welcome to WebmasterWorld Guest from 54.226.147.84
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
csci_b659/0.13
bull

10+ Year Member



 
Msg#: 3216 posted 6:04 am on Apr 18, 2006 (gmt 0)

156.56.103.16 - - [17/Apr/2006:22:24:29 -0500] "GET /robots.txt HTTP/1.1" 206 4646 "-" "csci_b659/0.13"
156.56.103.16 - - [17/Apr/2006:22:24:29 -0500] "GET /foo/ HTTP/1.1" 403 - "-" "csci_b659/0.13"

IP from Indiana University. Fetches ODP-listed directories, reads robots.txt, but which user-angent shall we list there?

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3216 posted 1:03 am on Apr 19, 2006 (gmt 0)

but which user-agent shall we list there?

Hey Jan,
How goes it?

Your likely one of a few to appreciate the humor in my answer?

"SetEnvIf User-Agent csci keep_out"

Don

bull

10+ Year Member



 
Msg#: 3216 posted 7:07 pm on Apr 20, 2006 (gmt 0)

Hey Don,

rhetorical question ;)
As many others, this one got caught by my anti-lowercase script. Couldn't live without it ;)

Cheers

Jan

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved