homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Obnoxious behavior from
Can't track it down...

 8:54 pm on Dec 6, 2000 (gmt 0)

IP made about 200 hits to my site yesterday, in 30 second bursts of 30-50 pages each. It fetched the same pages repeatedly in each visit. I came in to work today, and found the SAME visitor with the same behavior on the stats again...

I can't find ANY info on this IP... the OS is undefined, it doesn't seem to register any concrete browser info, and there's no referrer. A traceroute traced it back to it's own IP... no name on the server.

Here's a clip of one of it's hits from my access logs: - - [06/Dec/2000:07:22:59 -0900] "GET //main.html HTTP/1.1" 200 2848 "-" "Mozilla"

Does anyone have ANY idea who this obnoxious little beast is? Is it someone important, or should I just ban it from my site?



 10:43 pm on Dec 6, 2000 (gmt 0)

Well... I hunted some more, found an email address, and got this reply (from the company that controls the IP block containing the offender):

"I have notified the client, Employon.com, and they have stopped this activity. Employon.com is developing spiders which search for companies that post their jobs on the internet, and I'm told that they inadvertently
launched an errant spider, which was operating on a domain database that included your domain. Please let me know if this behavior persists, and I will take additional action.

Digibahn Tech Support"

If you see any super obnoxious 63.X IPs spidering your sites, and reverse IP lookup points to digibahn.com, it's probably the same folks (Employon.com)... But Digibahn is very responsive to the problem!


 2:26 am on Dec 7, 2000 (gmt 0)

Check out this link.
[Employon.com ]
and [beta.grassisgreener.com ]
Looks like they have a whole class C
Employon.com LLC (NETBLK-UU-63-86-155)
22700 Shore Center Drive
Euclid, OH 44123

Netname: UU-63-86-155
Netblock: -

If they do it again I'd go to grassisgreener.com and right a letter claiming that they are doing a DOS attack. They'll be nice to you after that and exclude your IP out of their crawling.


 2:52 am on Dec 7, 2000 (gmt 0)

Yeah, Employon/Grassisgreener seems to have their very own chunk of UUnet IPs... but a Digibahn email is listed as the "coordinator" under Employon's whois results, so that's who I wrote to.

I sent the same letter to grassisgreener.com... haven't heard back from them. Perhaps the NIC technical contact hadn't the foggiest idea what I was talking about?

Since Digibahn (the ones who seem to be "top dog" over the IP block in question) were so quick to respond, and offered to "take further action" if necessary, I'll probably go to them first if the spider is back tomorrow and just Cc: a copy of the message to grassisgreener.com If Digibahn is in the position to stomp on grassisgreener, they're the ones I want to talk to.


 10:36 am on Dec 7, 2000 (gmt 0)

Hi mivox - welcome to the world.

That is some very interesting information. A couple months ago, we had a question come up about posting SEO job information here on WebmasterWorld. You just confirmed we were correct in not allowing it to be posted for copyright concerns.


 6:46 pm on Dec 7, 2000 (gmt 0)

Yeah, grassisgreener's spider didn't show much respect for my site... but in all fairness, that particular spider may have still been under development. Their other spiders may observe more respected behavior protocols (asking for a robots.txt, timing page requests more moderately, etc.).

Funny, I always thought that the most obnoxious bots were generally looking for email addresses... now it's job listings.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved