Weird Unidentified Spider

[Samnwb]

Hi, i was looking for abit of enlightenment, ive had a funny spider hitting my site for the last 5 days on the trott, the first day it crawled it accessed nearly every accessible URL, but for the last 4 days it has done nothing but hit www.example.com//, why would it been hitting on my domain with 2 forward slashes attached?, that URL doesnt even exsist and it just sends you back to the primary domain address.

Details,

IP: 85.25.147.*
Agent String: Spider
Browser: Unknown
Hostname: hotel*.server4you.de
Whois: SERVER4YOU Dedicated Server Hosting, www.server4you.de

Has anyone else come across this sipder before?, do you think that i should be blocking it?

Regards samantha.

[edited by: incrediBILL at 2:34 am (utc) on June 28, 2008]
[edit reason] Obscured IPs [/edit]

[incrediBILL]

Hi Samantha and welcome to WebmasterWorld. The hosting company you refer to has had a few bots in the same IP block 85.25.147.* and I would simply ban them all.

85.25.147.* ""
85.25.147.* "MELBOT"
85.25.147.* "Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)"
85.25.147.* "Spider"

The "Spider" also keeps hitting my home page but it's blocked from entering so I'm not sure what they're after.

Note something in that range tried to incorrectly spoof MSIE 6 as well.

Nothing good going on best I can tell and I have the whole hosting company blocked.

[Samnwb]

Many thanks for your swift reply incrediBILL, on your recommendation i will now block this spider, btw im sorry for putting you to the trouble of editing my first post, i didn't realize that i was ment to mask the the last octet of the spider IP.

Regards samantha.

[edited by: Samnwb at 6:54 am (utc) on June 28, 2008]

[jdMorgan]

> I have the whole hosting company blocked

Note that many of us block the entire IP address range of that hosting company, not just the user-agent string or IP address of that one spider.

The main reason that another server would make requests to your server is if there is a site on that server that links to your site and it occasionally runs a server-side link-checker to see if the page that it links to on your site still exists. By keeping track of who links to your site, you should be able to block hosting company IP address ranges, yet still allow a few specific sites within that address range to access your server.

Jim

[wilderness]

Note that many of us block the entire IP address range of that hosting company, not just the user-agent string or IP address of that one spider.

At least one of "us" blocks the entire Class A (save ten numbers in the Class D). ;)

[Samnwb]

After a little more random investigation it would appear that the above mentioned spider in my first post might well be coming from www.artviper.net, one of the directories that my website is listed in is using website thumbnails from www.websitethumbnail.de, which is also part of www.artviper.net, that would explain why im getting visits from this sipder, but it still doesn't explain the random non existing URL crawling that this spider has been doing.

Regards samantha.

[edited by: Samnwb at 10:09 pm (utc) on July 3, 2008]

[incrediBILL]

Thanks for pointing me in their direction as I was able to get one of the tools on their website to generate easily identifiable traffic from the same IPs and then some.

They have a WEBCHECK tool that comes from 85.25.147.* with a blank UA and it in turn runs the W3C validator:

128.30.52.36 - "Jigsaw/2.2.5 W3C_CSS_Validator_JFouffa/2.0"

128.30.52.13 - "W3C_Validator/1.575"

Then it filled in the blanks here at the end with the names in the referrer instead of the UA:

85.25.147.* "Spider"
85.25.147.* "'artviper"

Case closed, this bot is solved.

[edited by: incrediBILL at 10:49 pm (utc) on July 3, 2008]