Forum Moderators: DixonJones

Message Too Old, No Replies

PayPal spider strikes again

it does not obey robots.txt

         

amznVibe

5:09 pm on Jan 17, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



After getting an alert from a site trapdoor with the ip of 65.206.229.143

I found this thread about the PayPal spider [webmasterworld.com...] (spiderman.nix.paypal.com)

looks like its doing what amazon's spider did early on, it does NOT obey robots.txt
it even tripped every form submission on the site, sheesh

In addition it does not have reverse DNS, REFER, or a user agent.

any reason why I should un-ban this spider?
they aren't doing anything useful that I can tell other than eating bandwidth at our expense

should we let PayPal know?

[edited by: amznVibe at 5:32 pm (utc) on Jan. 17, 2003]

amznVibe

5:13 pm on Jan 17, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Looks like they sent me this email immediately afterwards too:
..........
Welcome to PayPal Shops! Your website has now been registered and
will be listed in PayPal Shops within the next 24 hours, opening
the doors of your Shop to millions of PayPal members. We're
pleased that you're giving us the opportunity to help your online
business reach its full potential.
..........

I did register the site for PayPal Shops, but that doesn't give them the right to eat all that bandwidth and disobey robots.txt!