Internetseer

How do I kill this parasite?

[epanastatis]

Over the last week I've noticed a lot of hits from user-agent sitecheck.internetseer.com. I just added them to my robots.txt to ban them completely, because I never requested this service. In your experience, do they tend to obey the robots.txt or is there something more I need to do get rid of this thing?

[Macguru]

Hi epanastatis

You get "automagically" subscribed to internetseer.com when you use some online bulk submitters.

I believe you can get rid of them at the bottom of every email they will send you.

[mack]

Yep you can opt out of any future contact by following a link on their email. This also prevents them spidering you.

Mack.

[wilderness]

they've been spidering my sites massively for the past few days, even though all they are getting is 403's and they have not been allowed at my sites in over two years.

[claus]

Hmm.. bulk-emailers are not their only source for new sites to "monitor" then.

I suspect it's some creative form of marketing; follow top-something in the serps and start to "monitor" these in the hope that some logfile reading person will think it's a good idea to pay them to continue. They'll never get a penny/dime/eurocent from me.

[sidyadav]

they've been spidering my sites massively for the past few days, even though all they are getting is 403's and they have not been allowed at my sites in over two years.

How can they spider your sites when InternetSeer is only a Site checker service which checks your site/s and gives a report of every time your site is down etc. and at the end of the week e-mails you of how many times your site was down, how much time it took to connect everyday etc.

If you ask me, its a pretty handy tool.

Sid

[wilderness]

If you ask me, its a pretty handy tool.

Sid
I'm glad for you to have found a tool which is both useful and of benefit to you.

Tools are only handy if you desire to use them :)
I do not desire this tool visiting my websites.
or their email spams in my mailboxes.

Any visitor who crawls my pages in any capacity is spidering. Regardless of the software or method they use. Or even of their intent of use for what they are attempting to gather. My only concern is that a violation of my TOS has occurred and they are not a visitor which I desire.

The difference between an actual visitor and "something" crawling is in most instances very obvious in the logs.

[dmorison]

InternetSeer are the biggest logfile spammer out there. Period.

If you want their service that is fair enough; but the lame excuse you read at sitecheck.internetseer.com - the URL webmasters are referred to (and reads like marketing spiel) - is almost as pathetic an attempt at trying to legitimise their activities as those web hosting companies that try and make it look like your domain was entered onto their whois form.

InternetSeer is conducting an ongoing Survey that checks the connectivity of the Web. This information will be available to InternetSeer Subscribers and all other companies who have been a part of the Survey.

LOL

[epanastatis]

Checking my referral logs from yesterday, I got 10 visits from Internetseer. Less than the 18 from the day before, which may be accounted for by the fact that I didn't update my robots.txt to disallow it until the middle of that day. I'll check today's logs tomorrow and hopefully it will have disappeared.

I never signed up for this "service" and I haven't been receiving any e-mails from them. I'll check with my clients to see if they've been getting the e-mails instead.

[kevinpate]

Doesn't it check and obey a proper listing in your robots.txt?

I stuck a line in robots.txt some time back and I don't recall having seen any visits in a long time now.

[carfac]

I have just recently begun exploring a firewall built into FreeBSD (my web server os). Takes a little getting used to, but this is THE WAY to block those you do not want on your site. MUCH faster than the .htaccess or even the mod_perl blocks most of us use.

If you have your own server, and it is on FreeBSD, I suggest you explore IPFW. Here is a link to help you get started:

[freebsd.org...]

(Note this is a FREE program, I am not trolling for sales or anything!)

IPFW works at the kernel level, in fact, you will have to rebuild your kernel to use it. Also note, you better be local to the machine- it defaults to deny all!

Good luck!

dave

[epanastatis]

Well, I just checked my logs from yesterday and it appears that Internetseer is not abiding by my robots.txt exclusion. I got 7 visits from it yesterday and it kept treading over the same pages it always does. Since I'm quite the newbie, let me make sure my syntax is right:

User-agent: sitecheck.internetseer.com
Disallow: /

Since my website is hosted on a commercial server, what additional steps can I take to protect my logfiles from this bot?

I checked with my clients and it does not appear that anyone connected with the site in question has received an e-mail from these unscrupulous characters, so I'm a bit at a loss as to what to do next.

Thanks for your help so far.

[anchordesk]

Over the last six months, they have yet to access my robots.txt ... where they are banned. I've set up automatic blocking and bounceback for all their emails. I've emailed them directly trying to opt-out of their service which I never requested. Still, they are a regular visitor doing a head request of the same files over and over and over. My only solution has been to stop them at the door with htaccess.

[skipfactor]

epanastatis, according to their Website, you have the proper syntax.