homepage Welcome to WebmasterWorld Guest from 54.198.46.115
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Website Explorer
Website Explorer/0.9.9.9
Dijkgraaf




msg:4202985
 9:00 pm on Sep 16, 2010 (gmt 0)

UA: Website Explorer/0.9.9.9

IP: 110.67.109.NNN (so-net.ne.jp)

Robots.txt: No

Behaviour: Fetched a single page that is the home page of a webring, so there is the possibility of a 302 redirect confusing it, however the Referrer is the page itself.

 

Pfui




msg:4203035
 10:42 pm on Sep 16, 2010 (gmt 0)

Saw it a long time ago, in August, October, and December, 2008. Japan-based, too, like your hit. Blockworthy then, and now.

p1243-ipbf90*osakakita.osaka.ocn.ne.jp
Website Explorer/0.9.9.9

zv21814*.ppp.dion.ne.jp
Website Explorer/0.9.9.9

kd05913523711*.ppp.dion.ne.jp
Website Explorer/0.9.9.9

robots.txt? NO

jdMorgan




msg:4203249
 2:43 pm on Sep 17, 2010 (gmt 0)

I've blocked that UA for so long that it may be one of the five oldest specific-UA-blocks I've still got (I switched primarily to whitelisting UAs years ago).

Self-referring page requests are often a clue that it's a 'bot trying to be stealthy. I block such requests under certain circumstances.

But you have to be careful about blocking self-referrers if the referring page contains any "named anchors" because most user-agents (excepting Safari) will not send the URL-fragment used for named anchors.

So if for example you have a link like <a name="this-page's-URL-path#TOP">Back to Top</a> link on the page, then clicking this link can trigger a self-referred request on non-cacheable/short-expiry-time pages, if the visitor has browser caching disabled, if he/she flushes the browser cache after initially loading that page, or if he/she (somewhat erroneously) uses middle-click to open that link in a new tab (in MSIE), and you almost certainly don't want to block those requests.

Jim

thetrasher




msg:4203339
 5:05 pm on Sep 17, 2010 (gmt 0)

most user-agents (excepting Safari) will not send the URL-fragment used for named anchors
RFC 2616, 14.36 Referer: "The URI MUST NOT include a fragment."
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved