| 10:26 pm on Oct 5, 2000 (gmt 0)|
Hi Hope, I like your nick
We had a similar thread here [webmasterworld.com] - some of the bots on your list are in there.
DIIbot -> [digital-integrity.com ]
"Digital Integrity's patent-pending technology is the solution for
discovering and tracking all types of digital content. The power of
this technology is its ability to detect and track digital "content
segments" -- such as a Microsoft Word document or document
phrase, a PowerPoint slide, a JPEG file, a line of code, HTML, or
even MP3. But unlike a search engine, Digital Integrity's
technology can discover content segments of any length, in any
I don't know what GenCrawler, GentleSpider, Kenjin Spider and oBot ((compatible;Win32)) are - they are probably personal applications. I seem to recall a perl script Bot that was called 'GentleSpider'. If you have the spiders IPs we might be able to do some more tracking.
| 11:03 pm on Oct 5, 2000 (gmt 0)|
I was really hoping you wouldn't tell me to find the IP. I really hate looking at raw logs. I have heard they can make you go blind. ;) I will take a look and do a search on the IP.
| 11:25 pm on Oct 5, 2000 (gmt 0)|
Odds are good oBot ((compatible;Win32)) is some type of desktop app. The thing about User_Agent is that anyone with a bit of skill could hit your site using any UA he/she wants - so the name it self doesn't mean much.
| 11:49 pm on Oct 5, 2000 (gmt 0)|
You really know how to make my evening littleman. I was hoping they were all se spiders. It makes me nervous to think someone is spidering our entire site that many times. I wish I knew why. Oh well, guess I should go take a look at the logs to see what they were after and if they took a look at robots.txt.
| 2:56 am on Oct 6, 2000 (gmt 0)|
That's a good idea Hope, I'd be interested in what you find.
| 1:10 pm on Oct 9, 2000 (gmt 0)|
oBot is very polite. First thing it got was the robots.txt.
The IP for oBot is 188.8.131.52.
The other questionable bots listed above did not look for robots.txt.