1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 5:49 am Aug 4, 2026

Forum Moderators: open

Message Too Old, No Replies

johnhew crawler

         

keyplyr

9:22 pm on Feb 25, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




UA: johnhew crawler, johnhew@seas.upenn.edu/Nutch-1.12
Protocol: HTTP/1.1
Robots.txt: Yes
Host: AWS
54.160.0.0 - 54.175.255.255
54.160.0.0/12
54.224.0.0 - 54.239.255.255
54.224.0.0/12

lucy24

3:32 am on Feb 26, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Penn’s got at least three IP ranges. The robot couldn’t use one of them?

keyplyr

3:52 am on Feb 26, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Might be cheaper at AWS. Free account and only pay for data storage/usage.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 5:49 am Aug 4, 2026