Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

How to identify and track down spiders

6:44 pm on Jun 2, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 12

I get about one question every day about how to track down spider owners. Here is my method:

1) find the ip address.
2) do a trace route back to the host. For windows, go >start >run and type in tracert ipaddress. With ip address being the ip address of the spider.
3) let the trace finish. Notice back from the end of the trace, the last host you can find. Often this is a tricky step of deciding which was the last real host. Start at the bottom and work up. Usually you'll see if a host has 2-3 boxes and can determine the real host name by guessing.
4) take the host name and try finding it in the browser with some standard incantations of www.host.com or .net. Often that may be all you need.
5) look up the host on a internic whois. Often that can lead you straight to the owner/domain.

You can hit about 50% of them with this system. Most often you'll run into 'joe user' running a spider. Those are hard to know just who or what it was. If the spider was abusive, keep your logs and contact the admin of the host.

Most of the better isp's will take a moment to look into it - it may be someone who is routinely abusive and they need more information to identify them.

Anyone else with tips/tricks or comments on id'ing spiders?

2:25 pm on June 7, 2000 (gmt 0)

New User

10+ Year Member

joined:July 27, 2004
votes: 0

I don't know if it's the network I'm on, or what but I tried tracing this one and it times out on the second hop. It didn't come up looking under whois either. I'm really curious, though, cuz it's the first spider to crawl my ENTIRE site, start to finish. Got every BL page and all. Maybe someone else may have an idea on it:



4:22 pm on June 7, 2000 (gmt 0)

Inactive Member
Account Expired


Cool , Tx for this great tip!


1:11 am on June 8, 2000 (gmt 0)

Inactive Member
Account Expired



that IP belongs to http://www.seventwentyfour.com/
it looks like they have a link rot service ....

2:35 am on June 8, 2000 (gmt 0)

Full Member

10+ Year Member

joined:Sept 23, 1999
votes: 0

it looks like they have a link rot service

Link rot's the word! LOL

1:07 pm on June 8, 2000 (gmt 0)

New User

10+ Year Member

joined:July 27, 2004
votes: 0

Disappointed but relieved I guess. That was driving me nuts. I did get an email from them this morning saying there was a broken link on one of the BL pages. I forget which one now....email is on pc at home. I can send an email tonite with the broken link if you like? Thanks for solving that one BTW!

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members