| 8:06 pm on Nov 28, 2008 (gmt 0)|
Seen 3 times on 2 sites since Nov 22 - thanks!
| 3:03 am on Nov 29, 2008 (gmt 0)|
|You can easily thwart them blocking Peer1: |
OrgName: Peer 1 Dedicated Hosting
NetRange: 188.8.131.52 - 184.108.40.206
Any reason to not block all of Peer 1?
| 3:06 am on Nov 29, 2008 (gmt 0)|
|Any reason to not block all of Peer 1? |
Considering I host on Peer1/ServerBeach, I have to tread lightly with that.
| 3:20 am on Nov 29, 2008 (gmt 0)|
Ooops! Of course didnít mean to put you in awkward situation!
When blocking a server IP range from a server hosting organization I tend to block all similar named ranges from that organization - on the basis they are also probably used for servers.
| 4:34 am on Nov 29, 2008 (gmt 0)|
|I tend to block all similar named ranges from that organization |
Same here in most cases.
No need to leave gaping holes in the fence.
| 5:07 am on Dec 4, 2008 (gmt 0)|
Got a message from someone at WordTracker saying they don't crawl. They claim it's a lateral search tool that looks for keywords on all of the pages returned from the original search.
Sounds like quibbling over semantics about what constitutes a crawl or not because allowing a SE to crawl a site doesn't mean giving authorization for any other automated task to access pages resulting from that crawl and subsequent search, then crawling those pages yet again without permission.
But that's a different argument for a different day.
Anyway, they claim if you write to them they'll remove your site from their searches.
IMO, honoring robots.txt would certainly be a lot simpler for all involved.
| 7:03 pm on Dec 4, 2008 (gmt 0)|
We had a similar situation on one of the sites few month ago and wrote to WordTracker. They replied that their customer was doing a research using their services and they had no control over it. Few of the requests from it was made to an URI that contained no WWW. in it and contained "/..." as well. The only place that URI was reference ever was in MSN SERP: "host.tld/dir/page.h....". Attempts like that dated back to April of 2007. Another IP they have used on several occasions is 220.127.116.11.
REQUEST HEADERS from 18.104.22.168:
User-Agent: POE-Component-Client-HTTP/0.65 (perl; N; POE; en; rv:0.650000)
Notice that the there is no trailing forward slash on the referer.