PayPal spider, let it in or not?

PayPal "spiderman" does not obey robots.txt

[amznVibe]

After getting an alert from a site trapdoor with the ip of 65.206.229.143
I found this thread about the PayPal spider [webmasterworld.com...] (spiderman.nix.paypal.com)

looks like its doing what amazon's spider did early on, it does NOT obey robots.txt

In addition it does not have a REFER, or a user agent.

'till I know better this stays in the .HTACCESS
SetEnvIf Remote_Addr ^65\.206\.229\.143$ ban

any reason why I should un-ban this spider?
they aren't doing anything useful that I can tell other than eating bandwidth at our expense

Looks like they sent me this email immediately afterwards too:
..........
Welcome to PayPal Shops! Your website has now been registered and
will be listed in PayPal Shops within the next 24 hours, opening
the doors of your Shop to millions of PayPal members. We're
pleased that you're giving us the opportunity to help your online
business reach its full potential.
..........
I did register the site for PayPal Shops, but that doesn't give them the right to eat all that bandwidth and disobey robots.txt!

[Brett_Tabke]

How many hits are you seeing? Are you talking crawler or not?

Do you have Password Management turned on via PayPal?

[amznVibe]

Oh it tried to crawl the site but the trapdoor cut it off a few pages later.

I've alerted PayPal. They have at least acknoledged it was them and supposedly will forward to the appropriate people.

The no user agent, no refer, no rDNS and not obeying robots.txt kinda adds up to some programmer being incredibly lazy or just not giving a darn (or both?)

I've actually noticed on alot of their little cgi interfaces there are minor bugs and improper documentation, so there is a perhaps a general apathy going on in that part of the company.

Not sure what you mean by password management, everything on PayPal has passwords? Trying signing up for the PayPal store and you'll see the same thing in a few weeks. I can dig into the logs and post if you really want to see.

[Brett_Tabke]

If you setup any type of pass back from PayPal, they double check the security of the scripts on your site to make sure nothing is left laying around.

[edited by: Brett_Tabke at 2:50 pm (utc) on Jan. 18, 2003]

[amznVibe]

Oh I see what you are saying now, I have had IPN going on that particular site for a long time... this was definitely related to the PayPal store application.

They saw like 3 pages, then were banned, and they accepted the site for the store anyway. Why even bother to spider if the bot is that dumb.