Welcome to WebmasterWorld Guest from 54.226.238.178

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Anonymous Visits from Hosting Companies

     
12:35 pm on Jun 16, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member aristotle is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Aug 4, 2008
posts:2942
votes: 182


I've been seeing quite a few bot-like visits from IP addresses associated with small web hosting companies. Does anyone know the purpose of these visits?
10:27 pm on June 16, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


You would have to give us the user agents of those bots and a list of their actions before we could give you any insights.

There are tons of scrapers and spammers hosted everywhere, often using compromised machines to avoid detection via fast flux IP changes.

It's crazy out there.
11:03 pm on June 16, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member aristotle is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Aug 4, 2008
posts:2942
votes: 182


Here's the Latest Visitors entry for a case I saw today. It just downloaded the bare page without images. I haven't saved any other cases, but think they're similar to this.
Host: 209.190.64.233
/
Http Code: 200 Date: Jun 16 11:45:17 Http Version: HTTP/1.1 Size in Bytes: 39902
Referer: -
Agent: Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+5.2)

Here is the IP information I found:
IP: 209.190.64.233
Hostname: e9.40.be.static.xlhost.com
ISP: eNET
Organization: XLHost.com
Services: None detected
Type: Corporate
Assignment: Static IP
Country: United States
State/Region: Ohio
City: Columbus

So I really don't know what it is, or its purpose.
11:41 pm on June 16, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2817
votes: 73


I blocked that whole XLHost range about a year ago when a scraper was stuck in a trap. It was from 209.190.3.218, not the same IP, but humans don't generally visit from hosting IPs while anonymous robots do. It is always best if in doubt to see what they were up to. In this case they were not asking for robots.txt
209.190.0.0 - 209.190.127.255
209.190.0.0/17 ENET-XLHOST
12:04 am on June 17, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 30, 2002
posts:4843
votes: 2


Possibly a VPS used as a proxy or VPN.
1:01 am on June 17, 2014 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6676
votes: 131


@ aristotle

I manually looking through my raw server logs many times, each and every day. I look up every questionable hit. If it is from a hosting company, colocation service, cloud, or anything aimed at business services... I block that IP range.

IMO none of the above have any valid reason to request files from my server.

Occasionally there is some collateral damage, like company employees surfing during work hours. Each webmaster has to fine tune these types of gray areas.
3:30 am on June 17, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


xlhost is eminently blockable, and mixed-ranging. Here's another of their Columbus, Ohio-based ranges:

64.79.85.200 - 64.79.85.207
64.79.85.200/29

Last month, "niki-bot" (that's the entire UA) ran from one of those IPs and slammed into 403s for five minutes straight.
5:46 am on June 17, 2014 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6676
votes: 131



xlhost is eminently blockable, and mixed-ranging. Here's another of their Columbus, Ohio-based ranges:

64.79.85.200 - 64.79.85.207
64.79.85.200/29

That XLHost range is actually:

64.79.64.0 - 64.79.95.255
64.79.64.0/19
3:36 pm on June 17, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


Thanks! I usually block the closest CIDR and go upstream as needed. But when it comes to iffy server farms, blocking the parent's probably a good first move.
9:34 pm on June 17, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


For this one (and others using similar broken UA's?
See this thread [webmasterworld.com ]

SetEnvIf User-Agent " ; " keep_out
SetEnvIf User-Agent " \( " keep_out
SetEnvIf User-Agent "; " keep_out
SetEnvIf User-Agent "\) ; " keep_out

or

RewriteCond %{HTTP_USER_AGENT} \)\ \) [OR]
RewriteCond %{HTTP_USER_AGENT} \ ;[\ ] [OR]
RewriteCond %{HTTP_USER_AGENT} \ \([\ ] [OR]
RewriteCond %{HTTP_USER_AGENT} ;\ [\ ] [OR]
RewriteCond %{HTTP_USER_AGENT} \)\ ;[\ ]
RewriteRule .* - [F]
6:32 am on June 18, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13218
votes: 348


So I really don't know what it is, or its purpose.

If it's a new, just-getting-started robot it probably doesn't have its own server yet. So any IP lookup will take you only back to the (shared) host's name.
6:18 pm on June 18, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3125
votes: 4


Major reasons for bots on multiple IPs, even from servers, include scraping and virus-implant.

Hits may come from broadband-based botnets, server-based botnets or server space rented for the occasion, although that may be more expensive and less fun than hiring a botnet.
1:52 pm on July 6, 2014 (gmt 0)

New User

joined:July 6, 2014
posts: 32
votes: 0


I'm also facing the same issue.
regardless of the robots.txt file, they keep visiting and they are wasting my bandwidth.
3:28 pm on July 6, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2817
votes: 73


Hi Dammy, welcome to the Forums. The robots.txt file does not have any control over what any robots can or will do, it is your wish list, as in, "This is what I want you to do". Not all robots even look at that file and some that do, go on to ignore it.

The place where you can set limits is your .htaccess file and there are many resources here in the Forums to help you learn how to do that. I suggest you look through this Forum's Library if you want to be able to make them stop.
2:57 pm on July 7, 2014 (gmt 0)

New User

joined:July 6, 2014
posts: 32
votes: 0


@not2easy
Thanks for the reply, i'm always getting into problems with .htaccess, anyway i'm going to look 4 that.
Thanks
12:39 pm on July 16, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member aristotle is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Aug 4, 2008
posts:2942
votes: 182


Another small hosting company that shows up is Iliad Hosting in France. I found some information about it:
IP-range/subnet: 62.210.0.0-62.210.255.255
Description: IP Pool for Iliad-Entreprises Business Hosting Customers
Location: France (FR) flag
Registry: ripe
Number of domains hosted: 40,660
Number of nameservers hosted: 1,665
Number of mailservers hosted: 1,849
Number of SPAM hosts hosted: 120
Number of IP routes 2

Can someone give me advice on whether I should block this entire IP range. Also, what does it mean where it says that the number of "SPAM hosts" hosted is 120? Should I block the whole thing or not?
3:20 pm on July 16, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2817
votes: 73


"Business Hosting Customers" says you will never see a human visitor from that range. Whether to block or not is your call, I would if they were found "visiting".
4:52 pm on July 16, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member aristotle is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Aug 4, 2008
posts:2942
votes: 182


not2easy -- Thanks for the reply. But I just got a private email from Leosghost telling me that Iliad is also an ISP to the general French public. So I've decided not to block it out of concern that I could also block ISP users. There might be a way to differentiate, but I don't have the time, knowledge, or skill to make it worthwhile to pursue it any further.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members