Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- TosCrawler


lucy24 - 6:55 pm on Dec 25, 2012 (gmt 0)


:: bump ::

Now here's an interesting coincidence. After months of nibbling at a page here, a page there, Toshiba has taken to gulping up to 40 pages at once. Deep enough that I can be pretty sure they are honoring robots.txt. (I have two directories that are fully accessible to humans, but off-limits to robots.) Pages only, no other stuff.

We'll call it a coincidence because my e-mail IP is unrelated to my www IP, my signature never includes the domain name, and the log entry I quoted does not include a domain name. Quick detour to g### confirms that I appear to have the only site in the world with the exact pagename I randomly quoted. Oops.

Hmm.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4528638.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com