Forum Moderators: open

Message Too Old, No Replies

crawler3.wib.pa-x.dec.com ?

Anyone recognize this Spider?

         

24bit

5:51 am on Jul 18, 2002 (gmt 0)

10+ Year Member



It's probably obvious, but I can't figure out which spider this is. Any ideas?

william_dw

6:15 am on Jul 18, 2002 (gmt 0)

10+ Year Member



See:
[webresearch.pa-x.dec.com...] (it's a compaq spider for archive.org)

"If you are a person visiting this site, you are probably responding to a visit to your site by our web crawler, Mercator. We are a research laboratory, currently running web crawls on behalf of the Internet Archive. We are collecting many terabytes of data to add to the Archive's collection. We are not trying to spam your site or attack it in any way. We also do not intend to use your copyrighted material for any illegal purposes. Please see the Archive's privacy policy for more information about how they use the data in their collection and how to have your pages removed."

jdMorgan

6:16 am on Jul 18, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



24bit,
That's one of AltaVista's crawler domains. If you can see the User-agent in your raws logs, it
will probably be Scooter or Mercator. Mercator is the "checker" when you submit a site, I believe.
Scooter is the regular web-spidering crawler.
Jim

Brett_Tabke

6:22 am on Jul 18, 2002 (gmt 0)

24bit

4:33 am on Jul 19, 2002 (gmt 0)

10+ Year Member



Thanks! I sure hope it's Alta Vista, because it spidered my whole site. I don't have but a few pages in AV, and it would be great if they added my entire site.