Forum Moderators: open
looks like its doing what amazon's spider did early on, it does NOT obey robots.txt
In addition it does not have a REFER, or a user agent.
'till I know better this stays in the .HTACCESS
SetEnvIf Remote_Addr ^65\.206\.229\.143$ ban
any reason why I should un-ban this spider?
they aren't doing anything useful that I can tell other than eating bandwidth at our expense
Looks like they sent me this email immediately afterwards too:
..........
Welcome to PayPal Shops! Your website has now been registered and
will be listed in PayPal Shops within the next 24 hours, opening
the doors of your Shop to millions of PayPal members. We're
pleased that you're giving us the opportunity to help your online
business reach its full potential.
..........
I did register the site for PayPal Shops, but that doesn't give them the right to eat all that bandwidth and disobey robots.txt!
Do you have Password Management turned on via PayPal?
I've alerted PayPal. They have at least acknoledged it was them and supposedly will forward to the appropriate people.
The no user agent, no refer, no rDNS and not obeying robots.txt kinda adds up to some programmer being incredibly lazy or just not giving a darn (or both?)
I've actually noticed on alot of their little cgi interfaces there are minor bugs and improper documentation, so there is a perhaps a general apathy going on in that part of the company.
Not sure what you mean by password management, everything on PayPal has passwords? Trying signing up for the PayPal store and you'll see the same thing in a few weeks. I can dig into the logs and post if you really want to see.
[edited by: Brett_Tabke at 2:50 pm (utc) on Jan. 18, 2003]
They saw like 3 pages, then were banned, and they accepted the site for the store anyway. Why even bother to spider if the bot is that dumb.
