1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 2:37 pm May 4, 2026

Forum Moderators: open

Message Too Old, No Replies

newspaper

         

keyplyr

11:19 pm on Apr 5, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



UA: newspaper/0.0.9.8
Protocol: HTTP/1.1
Robots.txt: No
Host: Various

Simplified python article discovery & extraction. Coming from Google Cloud but could be run from anywhere.

keyplyr

9:39 pm on Jul 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




UA: newspaper/0.1.7
Protocol: HTTP/1.1
Robots.txt: No
Host: AWS
52.28.0.0 - 52.28.255.255
52.28.0.0/16
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. General SEO Issues / Crawler, Spider, and User Agent ID 2:37 pm May 4, 2026