Podtech Network crawler

Forum Moderators: open

Message Too Old, No Replies

Podtech Network crawler

Mokita

12:36 am on Jan 15, 2007 (gmt 0)

Failed to ask for robots.txt

Full UA: Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net)
IP: 71.134.252.nnn belonging to PPPoX Pool

GaryK

4:14 am on Jan 16, 2007 (gmt 0)

Hi Mokita. PodTech is essentially a feed aggregator. Normally all they do is grab an xml file and leave. I don't expect them to request robots.txt unless they're planning on doing something more than just taking my rss feed. What did it take from you?

Mokita

8:59 pm on Jan 16, 2007 (gmt 0)

Hi Gary,

Thanks - a search in Google before I made the OP told me what they are.

It only took the default home page, but the site has no rss feed available.

[edited by: Mokita at 9:00 pm (utc) on Jan. 16, 2007]

GaryK

7:36 pm on Jan 18, 2007 (gmt 0)

I wonder if they look at the home page to see if there's an rss link tag in the <head> section of the page and move on if nothing is found. I'm a lot more lax about bots not reading robots.txt if all they do is take my default root page. Then again they could have just been log spamming you. ;)

incrediBILL

1:11 am on Jan 24, 2007 (gmt 0)

I'm seeing a slightly differing UA:
71.134.252.#*$! "Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@turn.com)"

Wonder what's up with the different email address @turn.com and more importantly, if this was some commercial website aggregator, what in the heck is it doing using SBC's DSL lines from PPPoX Pool?

GaryK

7:39 pm on Jan 24, 2007 (gmt 0)

Being a podcast aggregator doesn't require much in the way of system resources. Just crawling each site you list to see if the feed has been updated. I can see someone doing that via their ISP.

As for the change in e-mail address, that's an interesting twist.

The first is a podcast aggregator, the second is a CPA ad network. Could this be part of a larger plan for Podtech Network? Well, Turn's How It Works page makes it seem they're more like AdWorks/AdSense than anything targeted at the podcast industry. So maybe not.

As we're so fond of stating: It bears further watching. :)