Forum Moderators: open
>Will spider using non stock User Agent. " How can I do? "
This means that the spider doesn't always use the name you would expect, like Scooter or Slurp, but instead, maybe it will use a name like Mozilla, or libwww-perl/5.48. This can make it harder to decide if it is a spider, or from which search engine it comes from.
>Will use IP Addresses from E X O D U S for spidering at times.
>"How can I do? What does E X O D U S mean?"
E X O D U S is a company that offers network outsourcing, some search engines use the network from E X O D U S to run their spiders at times. It makes it harder to dtermine if they are real spiders because the IP address is not from a familiar group of addresses, it comes from the range of addresses owned by EXODUS, and when the address is looked up, it does not show that it belongs to AltaVista, or Excite, instead it shows E X O D U S.
All of the exodus.net stuff can be found by using telnet to port 4321 on rwhois.exodus.net. Here is a sample (telnet) session for 216.34.109.92 which entered my site.
%rwhois V-1.5:003fff:00 rwhois.exodus.net (by Network Solutions, Inc. V-1.5.
3)
216.34.109.192
network:Auth-Area:216.34.0.0/16
network:Class-Name:network
network:Network-Name:216.34.109.0
network:IP-Network:216.34.109.0/24
network:Organization;I:Northern Light
network:Address-1;I:222 Third St. Suite 1320
network:Address-2;I:Cambridge , MA 02142
network:Created:99-AUG-11
network:Updated-By:saulin
%ok
I typed the ip above. I've also seen inktomi and av? (I think) using exodus.net.
HTH.
There's a couple of those rwhois utilities around, for those interested, here's one if command line is not your thing:
[rwhois.net...]
Very handy tool to keep in your toolbox.
Since you asked, startup, I'm using TeraTerm Pro to telnet and do other raw stuff, mainly for its secure shell (for Unix shell accounts). You can connect to any host on any port, SMTP, POP, HTTP, FTP, etc. (that will allow it of course).
One link to it is:
[zip.com.au...]
It's a Win9x or NT program, just type in the host: ie rwhois.exodus.net, and the port (4321) and connect. It will remember your connections. After connecting just type the ip and you get the raw output.
HTH.