| 10:59 pm on Nov 2, 2011 (gmt 0)|
Out of curiosity I just had a look at gluetext dot com, typed in two good keywords for one of my sites and their search engine (!?) began processing in real time. Where they get original URLs from I don't know (maybe one of the big three because there is no way that they can crawl the entire internet in under a minute) but they obviously did find my site and promptly started to rip through it (at least attempted to), in real time, while I was waiting at their site for the results to show.
Fortunately, I have a good system in place to prevent this from happening otherwise they would have gotten away with everything, now they got nothing.
UA : Mozilla/4.76 [en] (Win98; U)
It goes without saying, the whole range is now blocked !
| 1:06 am on Nov 3, 2011 (gmt 0)|
Thanks for the heads-up.
Had the RackSpce range reported by incrediBILL already blocked.
I put up the block on this new range, then searched for my top search terms. Competitors that are below me in the big 3 SERP showed up but not me.
188.8.131.52 - 184.108.40.206
or the wide deny 220.127.116.11/21 but I don't know what else is in there. Anyone?
| 1:46 am on Nov 3, 2011 (gmt 0)|
keyplyr could you see their attempt to access your site in your log files or do your log files record nothing when a range is blocked. Just curious
I blocked the whole
NetRange: 18.104.22.168 - 22.214.171.124
they all belong to the same server farm ;o)
| 2:29 am on Nov 3, 2011 (gmt 0)|
Yes my log files do show 403 responses when I block ranges, however I'll need to wait a couple more hours to review logs. I'll post again then.
| 11:12 am on Nov 3, 2011 (gmt 0)|
403'd total of 32 times, spaced out over 3 minutes.
Requested the top 3 web pages for the search terms I used. These would be the same web pages the big 3 SEs would list for these terms.
126.96.36.199 Mozilla/4.76 [en] (Win98; U)
| 7:10 pm on Nov 3, 2011 (gmt 0)|
Thank you keyplyr, I was just curious if their behaviour would be different when hitting a site that has blocked them (yours) and one that had not yet (mine).
After I entered my keywords I waited a moment while their progress thingy was running. Then I thought it was taking its time (in internet terms) to show the first results and on a hunch went to look what was happening at my site in the meantime and saw them trying to grasp page after page. Woh, I immediately closed the browser tap with their website and the their visit ended.
During that time they attempted to access 58 pages in about 1 min 46 sec.