Welcome to WebmasterWorld Guest from 54.196.144.100

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Spider list

     

Xoc

11:32 pm on Mar 6, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I thought I'd list the info from my logs ip addresses, user agents and domains for the major search engines in recent months. It also lists the first visit and most recent visit for each. There are some obvious blocks of ip addresses, but these are the ones that hit my server. It is in XML format, if you find that useful.

<spiders> 
<ip address="198.3.103.24" domain="voltaire.excite.com" ua="ArchitextSpider" dateFirst="2000-11-02" dateRecent="2001-03-01" />
<ip address="198.3.103.57" domain="kazoo.excite.com" ua="ArchitextSpider" dateFirst="2000-12-13" dateRecent="2001-02-05" />
<ip address="198.3.103.58" domain="kettle.excite.com" ua="ArchitextSpider" dateFirst="2000-10-31" dateRecent="2001-02-24" />
<ip address="198.3.103.59" domain="mandolin.excite.com" ua="ArchitextSpider" dateFirst="2000-11-20" dateRecent="2001-03-01" />
<ip address="198.3.103.60" domain="maraca.excite.com" ua="ArchitextSpider" dateFirst="2000-12-06" dateRecent="2000-12-06" />
<ip address="198.3.103.65" domain="piano.excite.com" ua="ArchitextSpider" dateFirst="2000-12-07" dateRecent="2001-02-07" />
<ip address="198.3.103.66" domain="piccolo.excite.com" ua="ArchitextSpider" dateFirst="2000-12-21" dateRecent="2001-02-25" />
<ip address="198.3.103.68" domain="ride.excite.com" ua="ArchitextSpider" dateFirst="2001-01-29" dateRecent="2001-02-04" />
<ip address="198.3.103.81" domain="viola.excite.com" ua="ArchitextSpider" dateFirst="2000-09-20" dateRecent="2001-01-27" />
<ip address="198.3.103.97" domain="maturana.excite.com" ua="ArchitextSpider" dateFirst="2001-03-01" dateRecent="2001-03-01" />
<ip address="199.172.149.129" domain="aloo.excite.com" ua="ArchitextSpider" dateFirst="2000-09-16" dateRecent="2001-02-01" />
<ip address="199.172.149.131" domain="bakra.excite.com" ua="ArchitextSpider" dateFirst="2000-12-02" dateRecent="2001-02-08" />
<ip address="199.172.149.135" domain="daal.excite.com" ua="ArchitextSpider" dateFirst="2000-10-31" dateRecent="2001-01-31" />
<ip address="199.172.149.138" domain="gajar.excite.com" ua="ArchitextSpider" dateFirst="2000-12-09" dateRecent="2001-03-02" />
<ip address="199.172.149.139" domain="halwa.excite.com" ua="ArchitextSpider" dateFirst="2000-12-15" dateRecent="2001-03-03" />
<ip address="199.172.149.141" domain="jeera.excite.com" ua="ArchitextSpider" dateFirst="2000-11-20" dateRecent="2001-03-03" />
<ip address="199.172.149.142" domain="kabab.excite.com" ua="ArchitextSpider" dateFirst="2000-11-21" dateRecent="2001-03-05" />
<ip address="199.172.149.143" domain="keema.excite.com" ua="ArchitextSpider" dateFirst="2001-02-24" dateRecent="2001-03-02" />
<ip address="199.172.149.144" domain="lassi.excite.com" ua="ArchitextSpider" dateFirst="2000-11-27" dateRecent="2000-11-27" />
<ip address="199.172.149.165" domain="abraham.excite.com" ua="ArchitextSpider" dateFirst="2000-12-22" dateRecent="2001-02-01" />
<ip address="199.172.149.166" domain="ariel.excite.com" ua="ArchitextSpider" dateFirst="2000-12-29" dateRecent="2001-02-01" />
<ip address="199.172.149.171" domain="diana.excite.com" ua="ArchitextSpider" dateFirst="2000-10-06" dateRecent="2001-02-01" />
<ip address="199.172.149.172" domain="duncan.excite.com" ua="ArchitextSpider" dateFirst="2000-11-09" dateRecent="2000-11-09" />
<ip address="199.172.149.176" domain="joseph.excite.com" ua="ArchitextSpider" dateFirst="2000-10-30" dateRecent="2001-01-31" />
<ip address="199.172.149.178" domain="macbeth.excite.com" ua="ArchitextSpider" dateFirst="2001-02-15" dateRecent="2001-02-15" />
<ip address="199.172.149.179" domain="macduff.excite.com" ua="ArchitextSpider" dateFirst="2000-12-28" dateRecent="2001-02-01" />
<ip address="199.172.149.180" domain="ophelia.excite.com" ua="ArchitextSpider" dateFirst="2000-12-22" dateRecent="2001-02-01" />
<ip address="199.172.149.182" domain="perdita.excite.com" ua="ArchitextSpider" dateFirst="2000-11-09" dateRecent="2001-02-01" />
<ip address="199.172.149.184" domain="philip.excite.com" ua="ArchitextSpider" dateFirst="2000-12-22" dateRecent="2001-02-01" />
<ip address="199.172.149.185" domain="romeo.excite.com" ua="ArchitextSpider" dateFirst="2000-12-29" dateRecent="2001-02-01" />
<ip address="199.172.149.187" domain="ursula.excite.com" ua="ArchitextSpider" dateFirst="2000-12-22" dateRecent="2001-02-01" />
<ip address="199.172.149.190" domain="bat.excite.com" ua="ArchitextSpider" dateFirst="2000-11-01" dateRecent="2001-01-31" />
<ip address="199.172.149.191" domain="gorilla.excite.com" ua="ArchitextSpider" dateFirst="2000-11-08" dateRecent="2000-11-08" />
<ip address="199.172.149.193" domain="leopard.excite.com" ua="ArchitextSpider" dateFirst="2000-12-28" dateRecent="2001-02-20" />
<ip address="199.172.149.195" domain="mantella.excite.com" ua="ArchitextSpider" dateFirst="2000-11-07" dateRecent="2001-02-20" />
<ip address="199.172.149.197" domain="ocelot.excite.com" ua="ArchitextSpider" dateFirst="2000-11-05" dateRecent="2001-01-31" />
<ip address="199.172.149.198" domain="panda.excite.com" ua="ArchitextSpider" dateFirst="2001-01-31" dateRecent="2001-02-01" />
<ip address="199.172.149.20" domain="orlando.excite.com" ua="ArchitextSpider" dateFirst="2001-02-25" dateRecent="2001-02-25" />
<ip address="199.172.149.205" domain="yapok.excite.com" ua="ArchitextSpider" dateFirst="2001-01-02" dateRecent="2001-02-01" />
<ip address="199.172.149.206" domain="gretzky.excite.com" ua="ArchitextSpider" dateFirst="2000-11-07" dateRecent="2001-02-20" />
<ip address="199.172.149.207" domain="hull.excite.com" ua="ArchitextSpider" dateFirst="2000-12-30" dateRecent="2001-01-31" />
<ip address="199.172.149.215" domain="electra.excite.com" ua="ArchitextSpider" dateFirst="2000-10-30" dateRecent="2001-02-01" />
<ip address="209.202.148.52" domain="recrawler2.bos2.fastsearch.net" ua="FAST-WebCrawler/2.2-pre27+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2001-01-30" dateRecent="2001-01-30" />
<ip address="64.41.254.25" domain="" ua="FAST-WebCrawler/2.2-pre34+(Multimedia+Search)+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2000-09-27" dateRecent="2001-02-01" />
<ip address="209.202.148.24" domain="crawler14.bos2.fast-search.net" ua="FAST-WebCrawler/2.2-pre40+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2000-12-03" dateRecent="2001-02-05" />
<ip address="209.202.148.36" domain="crawler26.bos2.fast-search.net" ua="FAST-WebCrawler/2.2-pre40+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2000-11-02" dateRecent="2001-02-05" />
<ip address="209.202.148.37" domain="crawler27.bos2.fast-search.net" ua="FAST-WebCrawler/2.2-pre40+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2000-11-01" dateRecent="2001-02-05" />
<ip address="209.202.148.39" domain="crawler29.bos2.fast-search.net" ua="FAST-WebCrawler/2.2-pre40+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2000-11-11" dateRecent="2001-02-04" />
<ip address="209.202.148.15" domain="crawler5.bos2.fast-search.net" ua="FAST-WebCrawler/2.2-pre41+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2000-12-22" dateRecent="2001-02-23" />
<ip address="209.202.148.23" domain="crawler13.bos2.fast-search.net" ua="FAST-WebCrawler/2.2-pre41+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2001-02-23" dateRecent="2001-02-23" />
<ip address="209.202.148.32" domain="crawler22.bos2.fast-search.net" ua="FAST-WebCrawler/2.2-pre41+(crawler@fast.no;+http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)" dateFirst="2001-02-23" dateRecent="2001-02-23" />
<ip address="64.208.36.2" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.23" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.28" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.30" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.35" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.39" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.42" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-29" />
<ip address="64.208.36.44" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-29" dateRecent="2000-09-29" />
<ip address="64.208.36.45" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.48" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-29" />
<ip address="64.208.36.49" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-26" dateRecent="2000-09-27" />
<ip address="64.208.36.53" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.55" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.57" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.58" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-28" dateRecent="2000-09-28" />
<ip address="64.208.36.59" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.60" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.61" domain="crawler5.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-30" dateRecent="2000-09-30" />
<ip address="64.208.36.63" domain="crawler5.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.69" domain="crawler5.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-29" />
<ip address="64.208.36.7" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-30" />
<ip address="64.208.36.74" domain="crawler5.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-29" dateRecent="2000-09-29" />
<ip address="64.208.36.77" domain="crawler5.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.79" domain="crawler5.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="64.208.36.8" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-09-28" />
<ip address="64.208.37.46" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-10-26" dateRecent="2000-10-29" />
<ip address="64.208.37.48" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-10-26" />
<ip address="64.208.37.55" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-10-27" dateRecent="2000-10-27" />
<ip address="64.208.37.59" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-10-26" />
<ip address="64.208.37.63" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-10-03" dateRecent="2000-10-25" />
<ip address="64.208.37.65" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-10-26" />
<ip address="64.208.37.77" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-10-27" />
<ip address="64.209.181.52" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-27" dateRecent="2000-10-01" />
<ip address="64.209.181.53" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://googlebot.com/bot.html)" dateFirst="2000-09-26" dateRecent="2000-10-01" />
<ip address="216.239.46.10" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-02" />
<ip address="216.239.46.11" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-24" dateRecent="2001-03-03" />
<ip address="216.239.46.113" domain="crawl4.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-02" />
<ip address="216.239.46.15" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-28" dateRecent="2001-03-01" />
<ip address="216.239.46.150" domain="crawl5.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-28" dateRecent="2001-03-02" />
<ip address="216.239.46.168" domain="crawl6.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-02" />
<ip address="216.239.46.17" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-24" dateRecent="2001-03-02" />
<ip address="216.239.46.19" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-02" />
<ip address="216.239.46.2" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-28" dateRecent="2001-03-02" />
<ip address="216.239.46.23" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-02" />
<ip address="216.239.46.26" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-22" dateRecent="2001-03-02" />
<ip address="216.239.46.27" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-22" dateRecent="2001-03-02" />
<ip address="216.239.46.28" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-22" dateRecent="2001-03-02" />
<ip address="216.239.46.3" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-22" dateRecent="2001-03-02" />
<ip address="216.239.46.30" domain="crawl1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-22" dateRecent="2001-03-01" />
<ip address="216.239.46.32" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-02" />
<ip address="216.239.46.33" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-22" dateRecent="2001-02-22" />
<ip address="216.239.46.34" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-24" dateRecent="2001-03-01" />
<ip address="216.239.46.39" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-24" dateRecent="2001-03-01" />
<ip address="216.239.46.41" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-01" />
<ip address="216.239.46.42" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-02" />
<ip address="216.239.46.44" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-28" dateRecent="2001-03-02" />
<ip address="216.239.46.45" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-03" />
<ip address="216.239.46.47" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-03-01" dateRecent="2001-03-01" />
<ip address="216.239.46.53" domain="crawl2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-02-28" dateRecent="2001-03-01" />
<ip address="64.208.37.1" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-11-22" dateRecent="2000-12-22" />
<ip address="64.208.37.10" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-24" dateRecent="2001-01-25" />
<ip address="64.208.37.11" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-25" dateRecent="2001-01-26" />
<ip address="64.208.37.113" domain="crawler4.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-12-22" dateRecent="2001-01-25" />
<ip address="64.208.37.15" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-24" dateRecent="2001-01-25" />
<ip address="64.208.37.150" domain="crawler5.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-25" dateRecent="2001-01-25" />
<ip address="64.208.37.168" domain="crawler6.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-26" dateRecent="2001-01-26" />
<ip address="64.208.37.17" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-24" dateRecent="2001-01-26" />
<ip address="64.208.37.19" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-05" dateRecent="2001-01-25" />
<ip address="64.208.37.2" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2001-01-25" />
<ip address="64.208.37.23" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-26" dateRecent="2001-01-25" />
<ip address="64.208.37.26" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-12-13" dateRecent="2001-01-25" />
<ip address="64.208.37.27" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-24" dateRecent="2001-01-26" />
<ip address="64.208.37.28" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2001-01-24" />
<ip address="64.208.37.3" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-11-23" dateRecent="2001-02-01" />
<ip address="64.208.37.30" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2001-01-26" />
<ip address="64.208.37.32" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-24" dateRecent="2001-01-24" />
<ip address="64.208.37.33" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-12-22" dateRecent="2001-01-25" />
<ip address="64.208.37.34" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-12-27" dateRecent="2001-01-24" />
<ip address="64.208.37.36" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-11-19" dateRecent="2000-11-19" />
<ip address="64.208.37.39" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2001-01-26" />
<ip address="64.208.37.40" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-12-22" dateRecent="2000-12-22" />
<ip address="64.208.37.41" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2001-01-24" dateRecent="2001-01-29" />
<ip address="64.208.37.42" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2001-01-24" />
<ip address="64.208.37.44" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-26" dateRecent="2001-01-26" />
<ip address="64.208.37.45" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2001-01-25" />
<ip address="64.208.37.47" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-12-22" dateRecent="2001-01-25" />
<ip address="64.208.37.49" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-12-23" />
<ip address="64.208.37.5" domain="proxy.google.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-11-30" dateRecent="2000-11-30" />
<ip address="64.208.37.53" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-05" dateRecent="2000-12-23" />
<ip address="64.208.37.54" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-12-23" />
<ip address="64.208.37.56" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-04" dateRecent="2000-12-22" />
<ip address="64.208.37.57" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-12-22" dateRecent="2000-12-22" />
<ip address="64.208.37.58" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-12-23" />
<ip address="64.208.37.60" domain="crawler2.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-12-23" />
<ip address="64.208.37.61" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-12-22" dateRecent="2000-12-22" />
<ip address="64.208.37.69" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-12-24" />
<ip address="64.208.37.7" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-12-23" />
<ip address="64.208.37.74" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-12-22" />
<ip address="64.208.37.78" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-25" dateRecent="2000-12-23" />
<ip address="64.208.37.79" domain="crawler3.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-26" dateRecent="2000-12-24" />
<ip address="64.208.37.9" domain="crawler1.googlebot.com" ua="Googlebot/2.1+(+http://www.googlebot.com/bot.html)" dateFirst="2000-10-28" dateRecent="2000-12-22" />
<ip address="208.219.77.19" domain="scooby.northernlight.com" ua="Gulliver/1.3" dateFirst="2000-09-20" dateRecent="2001-03-02" />
<ip address="208.219.77.29" domain="marvin.northernlight.com" ua="Gulliver/1.3" dateFirst="2000-09-16" dateRecent="2001-03-05" />
<ip address="216.34.109.191" domain="" ua="Gulliver/1.3" dateFirst="2000-11-21" dateRecent="2001-02-17" />
<ip address="216.34.109.192" domain="" ua="Gulliver/1.3" dateFirst="2000-09-16" dateRecent="2001-03-01" />
<ip address="209.247.40.101" domain="" ua="ia_archiver" dateFirst="2000-10-05" dateRecent="2000-10-05" />
<ip address="209.247.40.102" domain="" ua="ia_archiver" dateFirst="2000-10-05" dateRecent="2000-10-18" />
<ip address="209.247.40.103" domain="" ua="ia_archiver" dateFirst="2000-09-29" dateRecent="2000-10-17" />
<ip address="209.247.40.165" domain="arc15.alexa.com" ua="ia_archiver" dateFirst="2001-03-01" dateRecent="2001-03-05" />
<ip address="209.247.40.166" domain="arc16.alexa.com" ua="ia_archiver" dateFirst="2001-03-01" dateRecent="2001-03-03" />
<ip address="209.247.40.167" domain="arc17.alexa.com" ua="ia_archiver" dateFirst="2001-02-23" dateRecent="2001-03-01" />
<ip address="209.247.40.168" domain="arc18.alexa.com" ua="ia_archiver" dateFirst="2001-02-26" dateRecent="2001-03-02" />
<ip address="209.247.40.169" domain="arc19.alexa.com" ua="ia_archiver" dateFirst="2001-02-08" dateRecent="2001-03-06" />
<ip address="209.247.40.170" domain="arc20.alexa.com" ua="ia_archiver" dateFirst="2001-02-11" dateRecent="2001-02-11" />
<ip address="209.247.40.171" domain="arc21.alexa.com" ua="ia_archiver" dateFirst="2001-02-08" dateRecent="2001-03-05" />
<ip address="209.247.40.172" domain="arc22.alexa.com" ua="ia_archiver" dateFirst="2001-02-10" dateRecent="2001-03-06" />
<ip address="209.247.40.173" domain="arc23.alexa.com" ua="ia_archiver" dateFirst="2001-02-11" dateRecent="2001-03-05" />
<ip address="209.247.40.174" domain="arc24.alexa.com" ua="ia_archiver" dateFirst="2001-02-11" dateRecent="2001-03-05" />
<ip address="209.247.40.175" domain="arc25.alexa.com" ua="ia_archiver" dateFirst="2001-02-23" dateRecent="2001-02-23" />
<ip address="209.247.40.183" domain="ai-209-247-40-183.alexa.com" ua="ia_archiver" dateFirst="2001-02-06" dateRecent="2001-02-06" />
<ip address="209.247.40.201" domain="ai-209-247-40-201.alexa.com" ua="ia_archiver" dateFirst="2000-12-17" dateRecent="2001-02-03" />
<ip address="209.247.40.202" domain="ai-209-247-40-202.alexa.com" ua="ia_archiver" dateFirst="2000-12-17" dateRecent="2001-02-03" />
<ip address="209.247.40.203" domain="ai-209-247-40-203.alexa.com" ua="ia_archiver" dateFirst="2000-12-17" dateRecent="2001-02-03" />
<ip address="209.247.40.204" domain="ai-209-247-40-204.alexa.com" ua="ia_archiver" dateFirst="2000-10-23" dateRecent="2001-02-03" />
<ip address="209.247.40.205" domain="ai-209-247-40-205.alexa.com" ua="ia_archiver" dateFirst="2000-12-06" dateRecent="2001-02-03" />
<ip address="209.247.40.208" domain="ai-209-247-40-208.alexa.com" ua="ia_archiver" dateFirst="2000-12-31" dateRecent="2000-12-31" />
<ip address="209.247.40.210" domain="ai-209-247-40-210.alexa.com" ua="ia_archiver" dateFirst="2000-10-31" dateRecent="2000-10-31" />
<ip address="209.247.40.212" domain="ai-209-247-40-212.alexa.com" ua="ia_archiver" dateFirst="2000-11-10" dateRecent="2001-01-07" />
<ip address="209.247.40.213" domain="ai-209-247-40-213.alexa.com" ua="ia_archiver" dateFirst="2000-11-20" dateRecent="2001-01-19" />
<ip address="209.247.40.214" domain="ai-209-247-40-214.alexa.com" ua="ia_archiver" dateFirst="2000-11-20" dateRecent="2001-02-03" />
<ip address="209.247.40.215" domain="ai-209-247-40-215.alexa.com" ua="ia_archiver" dateFirst="2000-11-20" dateRecent="2001-02-03" />
<ip address="209.247.40.216" domain="ai-209-247-40-216.alexa.com" ua="ia_archiver" dateFirst="2000-11-20" dateRecent="2001-01-27" />
<ip address="209.247.40.217" domain="ai-209-247-40-217.alexa.com" ua="ia_archiver" dateFirst="2000-11-20" dateRecent="2001-01-17" />
<ip address="209.247.40.218" domain="ai-209-247-40-218.alexa.com" ua="ia_archiver" dateFirst="2001-01-17" dateRecent="2001-01-17" />
<ip address="209.247.40.219" domain="ai-209-247-40-219.alexa.com" ua="ia_archiver" dateFirst="2000-10-31" dateRecent="2000-10-31" />
<ip address="209.247.40.220" domain="ai-209-247-40-220.alexa.com" ua="ia_archiver" dateFirst="2000-11-09" dateRecent="2001-01-17" />
<ip address="209.247.40.223" domain="ai-209-247-40-223.alexa.com" ua="ia_archiver" dateFirst="2001-02-01" dateRecent="2001-02-01" />
<ip address="209.247.40.226" domain="ai-209-247-40-226.alexa.com" ua="ia_archiver" dateFirst="2001-01-31" dateRecent="2001-01-31" />
<ip address="209.247.40.232" domain="ai-209-247-40-232.alexa.com" ua="ia_archiver" dateFirst="2001-02-01" dateRecent="2001-02-01" />
<ip address="209.247.40.234" domain="ai-209-247-40-234.alexa.com" ua="ia_archiver" dateFirst="2001-01-30" dateRecent="2001-01-30" />
<ip address="209.247.40.237" domain="ai-209-247-40-237.alexa.com" ua="ia_archiver" dateFirst="2001-01-31" dateRecent="2001-01-31" />
<ip address="209.247.40.241" domain="ai-209-247-40-241.alexa.com" ua="ia_archiver" dateFirst="2000-09-20" dateRecent="2001-01-19" />
<ip address="209.247.40.242" domain="ai-209-247-40-242.alexa.com" ua="ia_archiver" dateFirst="2001-01-19" dateRecent="2001-01-19" />
<ip address="209.247.40.245" domain="ai-209-247-40-245.alexa.com" ua="ia_archiver" dateFirst="2000-10-09" dateRecent="2001-01-19" />
<ip address="209.247.41.110" domain="" ua="ia_archiver" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="209.247.41.113" domain="" ua="ia_archiver" dateFirst="2000-09-20" dateRecent="2000-10-01" />
<ip address="209.247.40.243" domain="ai-209-247-40-243.alexa.com" ua="InternetArchive-1.0" dateFirst="2000-09-19" dateRecent="2000-09-19" />
<ip address="209.247.40.246" domain="ai-209-247-40-246.alexa.com" ua="InternetArchive-1.0" dateFirst="2000-10-01" dateRecent="2000-10-01" />
<ip address="199.172.148.105" domain="bunsen.excite.com" ua="libwww-perl/5.33" dateFirst="2001-03-06" dateRecent="2001-03-06" />
<ip address="206.79.171.130" domain="sjc-fe14-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-12-21" dateRecent="2001-03-04" />
<ip address="206.79.171.156" domain="sjc-fe7-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-16" dateRecent="2001-03-04" />
<ip address="206.79.171.157" domain="sjc-fe13-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-10" dateRecent="2001-02-06" />
<ip address="206.79.171.16" domain="sjc-fe1-2.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-06" dateRecent="2001-02-15" />
<ip address="206.79.171.17" domain="sjc-fe2-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-12" dateRecent="2001-02-15" />
<ip address="206.79.171.19" domain="sjc-fe3-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-14" dateRecent="2001-02-15" />
<ip address="206.79.171.21" domain="sjc-fe4-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-21" dateRecent="2001-03-01" />
<ip address="206.79.171.23" domain="sjc-fe5-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-06" dateRecent="2001-03-05" />
<ip address="206.79.171.67" domain="sjc-fe6-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-11" dateRecent="2001-03-05" />
<ip address="206.79.171.81" domain="sjc-fe8-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-21" dateRecent="2001-03-04" />
<ip address="206.79.171.85" domain="sjc-fe9-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-07" dateRecent="2001-02-15" />
<ip address="206.79.171.89" domain="sjc-fe10-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-16" dateRecent="2001-02-15" />
<ip address="206.79.171.93" domain="sjc-fe11-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-06" dateRecent="2001-02-15" />
<ip address="206.79.171.97" domain="sjc-fe12-1.sjc.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-10" dateRecent="2001-03-03" />
<ip address="209.202.192.129" domain="bos-fe1-4.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-16" dateRecent="2001-03-02" />
<ip address="209.202.192.130" domain="bos-fe2-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-16" dateRecent="2001-02-06" />
<ip address="209.202.192.137" domain="bos-fe3-4.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-02-13" dateRecent="2001-02-28" />
<ip address="209.202.192.138" domain="bos-fe4-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-05" dateRecent="2001-03-02" />
<ip address="209.202.192.141" domain="bos-fe5-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-16" dateRecent="2001-02-28" />
<ip address="209.202.192.146" domain="bos-fe6-4.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-02-27" dateRecent="2001-03-03" />
<ip address="209.202.192.147" domain="bos-fe7-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-15" dateRecent="2001-02-24" />
<ip address="209.202.192.160" domain="bos-fe8-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-27" dateRecent="2001-02-23" />
<ip address="209.202.192.165" domain="bos-fe9-4.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-13" dateRecent="2001-02-25" />
<ip address="209.202.192.166" domain="bos-fe10-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-12-21" dateRecent="2001-02-24" />
<ip address="209.202.192.179" domain="bos-fe11-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-05" dateRecent="2001-02-09" />
<ip address="209.202.192.182" domain="bos-fe12-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-15" dateRecent="2001-03-06" />
<ip address="209.202.192.185" domain="bos-fe13-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-02-25" dateRecent="2001-02-27" />
<ip address="209.202.192.198" domain="bos-fe14-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-09" dateRecent="2001-02-22" />
<ip address="209.202.192.201" domain="bos-fe15-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-19" dateRecent="2001-01-19" />
<ip address="209.202.192.204" domain="bos-fe16-1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2001-01-09" dateRecent="2001-01-28" />
<ip address="209.202.193.32" domain="bos-bc02s01.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-09-28" dateRecent="2001-01-01" />
<ip address="209.202.193.34" domain="bos-bc02s03.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-10-02" dateRecent="2001-01-03" />
<ip address="209.202.193.35" domain="bos-bc02s04.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-15" dateRecent="2001-01-31" />
<ip address="209.202.193.37" domain="bos-bc02s06.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-12-28" dateRecent="2001-01-18" />
<ip address="209.202.193.38" domain="bos-bc02s07.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-09-28" dateRecent="2000-11-10" />
<ip address="209.67.229.100" domain="bos-spider9b.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-10-20" dateRecent="2000-12-18" />
<ip address="209.67.229.101" domain="bos-spider10b.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-28" dateRecent="2000-12-01" />
<ip address="209.67.229.138" domain="bos-spider1.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-12-19" dateRecent="2000-12-22" />
<ip address="209.67.229.139" domain="bos-spider2.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-07" dateRecent="2000-12-24" />
<ip address="209.67.229.140" domain="bos-spider3.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-12-23" dateRecent="2000-12-23" />
<ip address="209.67.229.142" domain="bos-spider5.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-03" dateRecent="2000-12-22" />
<ip address="209.67.229.143" domain="bos-spider6.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-10-24" dateRecent="2000-11-10" />
<ip address="209.67.229.144" domain="bos-spider7.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-10-09" dateRecent="2000-12-24" />
<ip address="216.35.194.183" domain="bos-spider13.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-12-08" dateRecent="2000-12-21" />
<ip address="216.35.194.184" domain="bos-spider14.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-17" dateRecent="2000-12-22" />
<ip address="216.35.194.185" domain="bos-spider15.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-10-31" dateRecent="2000-12-08" />
<ip address="216.35.194.186" domain="bos-spider16.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-11-30" dateRecent="2000-12-21" />
<ip address="216.35.194.188" domain="bos-spider18.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-12-07" dateRecent="2000-12-20" />
<ip address="216.35.194.189" domain="bos-spider19.bos.lycos.com" ua="Lycos_Spider_(T-Rex)" dateFirst="2000-12-04" dateRecent="2000-12-04" />
<ip address="204.123.28.10" domain="crawler0-complaints-to-admin.webresearch.pa-x.dec.com" ua="Mercator-1.0" dateFirst="2000-09-29" dateRecent="2001-02-14" />
<ip address="204.123.28.11" domain="crawler1.webresearch.pa-x.dec.com" ua="Mercator-1.0" dateFirst="2000-11-07" dateRecent="2001-02-12" />
<ip address="212.187.226.99" domain="ukaddurl02r.aveurope.co.uk" ua="Mercator-1.1" dateFirst="2001-02-26" dateRecent="2001-03-01" />
<ip address="204.123.28.30" domain="atrax0.pa-x.dec.com" ua="Mercator-2.0" dateFirst="2000-12-07" dateRecent="2000-12-07" />
<ip address="204.123.28.31" domain="atrax1.pa-x.dec.com" ua="Mercator-2.0" dateFirst="2000-10-07" dateRecent="2000-12-07" />
<ip address="204.123.28.32" domain="atrax2.pa-x.dec.com" ua="Mercator-2.0" dateFirst="2000-12-07" dateRecent="2000-12-25" />
<ip address="209.73.164.127" domain="tv34.sv.av.com" ua="Mercator-Scrub-1.1" dateFirst="2000-11-17" dateRecent="2001-02-14" />
<ip address="209.73.164.129" domain="tv35.sv.av.com" ua="Mercator-Scrub-1.1" dateFirst="2000-11-25" dateRecent="2001-02-17" />
<ip address="202.212.5.32" domain="goo311.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-11-28" dateRecent="2001-03-05" />
<ip address="209.131.48.140" domain="crusade.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-26" dateRecent="2001-01-02" />
<ip address="209.185.143.96" domain="q2000.inktomisearch.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-27" dateRecent="2001-02-09" />
<ip address="209.185.143.98" domain="q2002.inktomisearch.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2001-01-29" dateRecent="2001-02-06" />
<ip address="216.35.103.79" domain="si4000.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2001-03-03" />
<ip address="216.35.103.80" domain="si4001.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2001-03-03" />
<ip address="216.35.103.81" domain="si4002.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2001-03-03" />
<ip address="216.35.116.86" domain="wm3006.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-22" dateRecent="2001-03-06" />
<ip address="216.35.116.87" domain="wm3007.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2001-03-05" />
<ip address="216.35.116.90" domain="si3000.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2001-03-06" />
<ip address="216.35.116.91" domain="si3001.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2001-03-05" />
<ip address="216.35.116.92" domain="si3002.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2001-03-06" />
<ip address="216.35.116.93" domain="si3003.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2001-03-05" />
<ip address="216.35.116.97" domain="q5101.inktomi.com" ua="Mozilla/3.0+(Slurp/si;+slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-12-01" dateRecent="2001-03-03" />
<ip address="209.67.252.211" domain="" ua="Mozilla/4.06+[en]+(Direct+Hit+Grabber)" dateFirst="2000-12-08" dateRecent="2000-12-08" />
<ip address="209.67.252.212" domain="" ua="Mozilla/4.06+[en]+(Direct+Hit+Grabber)" dateFirst="2000-11-04" dateRecent="2000-11-04" />
<ip address="209.67.252.214" domain="" ua="Mozilla/4.06+[en]+(Direct+Hit+Grabber)" dateFirst="2000-10-16" dateRecent="2000-10-16" />
<ip address="209.67.252.215" domain="" ua="Mozilla/4.06+[en]+(Direct+Hit+Grabber)" dateFirst="2000-09-20" dateRecent="2000-09-20" />
<ip address="64.55.148.51" domain="" ua="Mozilla/4.06+[en]+(Direct+Hit+Grabber)" dateFirst="2001-01-16" dateRecent="2001-02-02" />
<ip address="64.55.148.52" domain="" ua="Mozilla/4.06+[en]+(Direct+Hit+Grabber)" dateFirst="2001-02-10" dateRecent="2001-02-10" />
<ip address="64.55.148.53" domain="" ua="Mozilla/4.06+[en]+(Direct+Hit+Grabber)" dateFirst="2001-01-11" dateRecent="2001-01-17" />
<ip address="64.55.148.54" domain="" ua="Mozilla/4.06+[en]+(Direct+Hit+Grabber)" dateFirst="2001-01-18" dateRecent="2001-01-18" />
<ip address="209.73.174.250" domain="avfwclient.sv.av.com" ua="roach.smo.av.com-1.0" dateFirst="2000-12-01" dateRecent="2000-12-13" />
<ip address="128.177.243.65" domain="" ua="Scooter/1.0" dateFirst="2000-09-27" dateRecent="2000-09-27" />
<ip address="209.73.164.13" domain="vscooter.sv.av.com" ua="Scooter/1.0" dateFirst="2000-10-25" dateRecent="2001-02-04" />
<ip address="209.73.164.16" domain="mmbuild2.sv.av.com" ua="Scooter/1.0" dateFirst="2001-01-17" dateRecent="2001-01-17" />
<ip address="209.73.164.40" domain="scooter.sv.av.com" ua="Scooter/1.0" dateFirst="2000-10-23" dateRecent="2001-03-06" />
<ip address="209.73.164.50" domain="bigip1-snat.sv.av.com" ua="Scooter/1.0" dateFirst="2000-11-16" dateRecent="2001-03-01" />
<ip address="128.177.243.83" domain="" ua="Scooter/2.0+G.R.A.B.+V1.1.0" dateFirst="2000-09-30" dateRecent="2000-10-02" />
<ip address="128.177.243.84" domain="" ua="Scooter/2.0+G.R.A.B.+V1.1.0" dateFirst="2000-09-29" dateRecent="2000-09-29" />
<ip address="209.73.164.43" domain="brillo.sv.av.com" ua="Scooter/2.0+G.R.A.B.+V1.1.0" dateFirst="2000-10-28" dateRecent="2000-10-28" />
<ip address="209.73.164.41" domain="scooter2.sv.av.com" ua="Scooter2_Mercator_3-1.0" dateFirst="2000-10-23" dateRecent="2000-12-02" />
<ip address="209.73.164.126" domain="tv33.sv.av.com" ua="Scooter-Jellyfish1" dateFirst="2000-11-18" dateRecent="2001-03-02" />
<ip address="209.73.164.42" domain="test-scooter.sv.av.com" ua="Scooter-test-scooter_Mc_kalyan_38" dateFirst="2000-10-25" dateRecent="2001-02-01" />
<ip address="209.73.164.130" domain="tv36.sv.av.com" ua="Scooter-tv36_Mc_conn_35" dateFirst="2000-11-25" dateRecent="2001-02-10" />
<ip address="209.1.13.231" domain="j300.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-16" dateRecent="2001-02-01" />
<ip address="209.1.13.232" domain="j301.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-19" dateRecent="2001-02-10" />
<ip address="209.131.62.107" domain="ec007.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-03" dateRecent="2000-10-03" />
<ip address="209.131.62.109" domain="ec009.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-02" dateRecent="2000-10-02" />
<ip address="209.67.206.126" domain="" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-30" dateRecent="2000-09-30" />
<ip address="209.67.206.127" domain="" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-21" dateRecent="2000-10-03" />
<ip address="216.35.103.51" domain="j4011.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-25" dateRecent="2000-12-07" />
<ip address="216.35.103.52" domain="j4012.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-18" dateRecent="2000-11-22" />
<ip address="216.35.103.53" domain="j4013.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-19" dateRecent="2000-11-27" />
<ip address="216.35.103.54" domain="j4014.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-18" dateRecent="2000-12-03" />
<ip address="216.35.103.55" domain="j4015.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-24" dateRecent="2000-11-20" />
<ip address="216.35.103.56" domain="j4016.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2000-11-21" />
<ip address="216.35.103.57" domain="j4017.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-19" dateRecent="2000-12-06" />
<ip address="216.35.103.58" domain="j4018.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2000-11-16" />
<ip address="216.35.103.59" domain="j4019.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2000-12-06" />
<ip address="216.35.103.60" domain="j4020.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-24" dateRecent="2000-11-20" />
<ip address="216.35.103.71" domain="j110.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-16" dateRecent="2001-02-22" />
<ip address="216.35.103.72" domain="j111.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-19" dateRecent="2001-03-02" />
<ip address="216.35.103.74" domain="j4021.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2000-11-07" />
<ip address="216.35.103.75" domain="j4022.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-24" dateRecent="2000-11-16" />
<ip address="216.35.103.76" domain="j4023.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-18" dateRecent="2000-12-03" />
<ip address="216.35.103.77" domain="j4024.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-04" dateRecent="2000-12-03" />
<ip address="216.35.103.78" domain="j4025.inktomisearch.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-18" dateRecent="2000-12-03" />
<ip address="216.35.116.41" domain="j3001.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2000-11-28" />
<ip address="216.35.116.42" domain="j3002.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2000-11-28" />
<ip address="216.35.116.43" domain="j3003.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-28" dateRecent="2000-11-10" />
<ip address="216.35.116.44" domain="j3004.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-29" dateRecent="2000-11-06" />
<ip address="216.35.116.45" domain="j3005.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2000-11-21" />
<ip address="216.35.116.46" domain="j3006.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-11-15" dateRecent="2000-11-28" />
<ip address="216.35.116.47" domain="j3007.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2000-11-19" />
<ip address="216.35.116.48" domain="j3008.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-11-07" dateRecent="2000-11-20" />
<ip address="216.35.116.49" domain="j3009.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-31" dateRecent="2000-11-17" />
<ip address="216.35.116.50" domain="j3010.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-31" dateRecent="2000-11-24" />
<ip address="216.35.116.51" domain="j3011.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-11-07" dateRecent="2000-11-14" />
<ip address="216.35.116.52" domain="j3012.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-29" dateRecent="2000-11-27" />
<ip address="216.35.116.53" domain="j3013.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-28" dateRecent="2000-11-11" />
<ip address="216.35.116.54" domain="j3014.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-19" dateRecent="2000-11-13" />
<ip address="216.35.116.55" domain="j3015.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-30" dateRecent="2000-11-14" />
<ip address="216.35.116.56" domain="j3016.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-28" dateRecent="2000-11-18" />
<ip address="216.35.116.57" domain="j3017.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-28" dateRecent="2000-11-12" />
<ip address="216.35.116.58" domain="j3018.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-19" dateRecent="2000-11-24" />
<ip address="216.35.116.59" domain="j3019.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2000-11-28" />
<ip address="216.35.116.88" domain="j3020.inktomi.com" ua="Slurp.so/1.0+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-11-02" dateRecent="2000-11-16" />
<ip address="216.35.116.105" domain="wm3020.inktomi.com" ua="Slurp/cat+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-17" dateRecent="2000-11-02" />
<ip address="216.35.116.107" domain="wm3022.inktomi.com" ua="Slurp/cat+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-05" dateRecent="2000-10-05" />
<ip address="216.35.116.108" domain="wm3023.inktomi.com" ua="Slurp/cat+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-15" dateRecent="2000-10-03" />
<ip address="216.35.116.109" domain="wm3024.inktomi.com" ua="Slurp/cat+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-18" dateRecent="2000-10-02" />
<ip address="164.124.250.214" domain="cache.bora.net" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-15" dateRecent="2000-09-15" />
<ip address="164.124.250.220" domain="cache.bora.net" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-07" dateRecent="2000-10-07" />
<ip address="202.212.5.30" domain="panda.inktomi.com" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-19" dateRecent="2000-12-07" />
<ip address="203.255.234.106" domain="" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-23" dateRecent="2000-10-08" />
<ip address="209.131.48.219" domain="sqadev1.inktomi.com" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-29" dateRecent="2001-01-20" />
<ip address="209.67.206.133" domain="" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-19" dateRecent="2001-03-01" />
<ip address="211.169.241.21" domain="" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-28" dateRecent="2000-12-04" />
<ip address="213.216.143.37" domain="" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-04" dateRecent="2000-12-07" />
<ip address="213.216.143.38" domain="" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-11-02" dateRecent="2000-12-07" />
<ip address="213.216.143.39" domain="" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-15" dateRecent="2000-12-07" />
<ip address="216.35.103.61" domain="j100.inktomi.com" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-20" dateRecent="2001-02-07" />
<ip address="216.35.103.62" domain="j101.inktomi.com" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2000-10-04" />
<ip address="216.35.116.103" domain="wm3018.inktomi.com" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-10-01" dateRecent="2000-10-05" />
<ip address="216.35.116.106" domain="wm3021.inktomi.com" ua="Slurp/si+(slurp@inktomi.com;+http://www.inktomi.com/slurp.html)" dateFirst="2000-09-17" dateRecent="2000-10-03" />
</spiders>

littleman

1:44 am on Mar 7, 2001 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Wow, Xoc that is quite a list.
I'm sure some people are going to be happy to see it!

Xoc

2:29 am on Mar 7, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I wrote a tool for analyzing my log files to see spider traffic. It builds a database of this information, including doing the reverse IP address lookup. Another simple program filters all the spiders in the database to just the major ones and dumps into this XML format.

I use this list in my 404 handler to keep spiders from forwarding to the new page. I want the bloody spiders to drop the old urls, but I want people to get the page they asked for when they click on a link in the SERP.

I programatically define a spider as an IP address that has ever hit robots.txt, or doesn't have Mozilla or Opera in the user agent field.

skirril

7:32 am on Mar 16, 2001 (gmt 0)

10+ Year Member



pls add:
216.177.11.53
ua: /home/survey/bin/AMBotM1frame

tcmalaska

4:00 pm on Mar 16, 2001 (gmt 0)

10+ Year Member



XOC
I wouldnt put too much emphasis on the UA,
several I mean SEVERAL will come in with a mozilla ua.

"I programatically define a spider as an IP address that has ever hit robots.txt, or doesn't have Mozilla or Opera in the user agent field"

but the robots.txt is helpful, then watch patterns
like the mozilla ua that always follows right behind a listed spider etc, another thing that works is d alot of lookups and research on some of your "noreferrer-nohostname and no agent"
I personally lokk into alot of the "no referrer-no hostname-generic mozilla ua"

Ive nailed inktomi back when they first started using exodus just by accident.

but yes a good list
Todd

Xoc

2:05 am on Mar 24, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I look for two things: 1, I look for the appropriate string in the User Agent. But 2, I resolve all IP addresses that have ever hit robots.txt and if any of them come from the domain of the search engine, I list that.

I am doing an extremely limited sort of cloaking on my sites. I have a 404 handler to redirect hits to new locations when a file moves. But I want the search engines not to follow the redirect, but rather drop the old page from the index. So my list is meant to have the search engines get a real 404, not a redirect when they hit those files. If, for some reason, the engine gets through, I'll just get them to drop it next time they try to index. So I'm not too worried about catching every spider that comes by.