Welcome to WebmasterWorld Guest from 54.147.44.93

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Punk Spider

caught this punk snooping from the clouds

     

incrediBILL

1:22 am on Jul 8, 2012 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



184.106.80.222

"Punk Spider/PunkSPIDER-v0.1"

robots.txt: YES

wilderness

2:27 am on Jul 8, 2012 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



"Punk Spider"

Arent they all ;)

lucy24

3:01 am on Jul 8, 2012 (gmt 0)

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



v0.1

Oi! Save your alpha testing for your friends' sites! Come back when you're ready with v. ... well, OK, at least 0.8.

dstiles

6:29 pm on Jul 8, 2012 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Server range (Rackspace). Already blocked.

incrediBILL

6:42 pm on Jul 8, 2012 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Server range (Rackspace). Already blocked.


That too.

I attempt to be polite with robots.txt set aside as a special case in the firewall so the dynamic robots.txt code will serve up permissions to anyone that asks. However, if you ask for robots.txt and are denied, and then request any other page from that user agent or IP address, you're also denied since the script enforces the robots.txt rules.

The reason I track the IP is some smart ass started asking for robots.txt using one user agent to test the waters then switched the user agent when asking for pages so I started tracking the IPs making the robots.txt requests if they're denied :)

The data center blocking, which includes rackspace, applies to all other files :)

It's complicated yet so simple.

dstiles

9:21 pm on Jul 9, 2012 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Your system is far more sophisticated than mine. :)

My IIS system (sans htaccess) can only intercept IPs and headers on ASP page access. Far too late for me to change it now. :(

incrediBILL

7:23 am on Jul 21, 2012 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



That PUNK came from a new IP today: 198.101.170.228

Wondering if they're in the cloud.

dstiles

10:14 pm on Jul 21, 2012 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Don't care. Rackspace: blocked 198.101.128/17 :)

Does rackspace operate a cloud? If so, where? What IPs?
 

Featured Threads

Hot Threads This Week

Hot Threads This Month