Forum Moderators: DixonJones
I found this thread about the PayPal spider [webmasterworld.com...] (spiderman.nix.paypal.com)
looks like its doing what amazon's spider did early on, it does NOT obey robots.txt
it even tripped every form submission on the site, sheesh
In addition it does not have reverse DNS, REFER, or a user agent.
any reason why I should un-ban this spider?
they aren't doing anything useful that I can tell other than eating bandwidth at our expense
should we let PayPal know?
[edited by: amznVibe at 5:32 pm (utc) on Jan. 17, 2003]
I did register the site for PayPal Shops, but that doesn't give them the right to eat all that bandwidth and disobey robots.txt!