What you really want is Slurp.so, thats the deep spider which usually gets you into permanent database. Slurp/si methinks is just link checker and 404 cleaner but who knows with the new Inktomi 500mln index things may have changed.
drbill, yes I think you are led to believe ;) Slurp.so from j30xx.inktomi.com 216.32.116.yy xx-starts from 01 yy starts from 41 this is not japanese spider, it goes in many levels deep and I it used to be money, with new inktomi time will tell.
hey all :-) here is something I have noticed, any and all input is greatly appreciated. From my log files; 1) Sat Sep 2 18:46:02 2000 --¦Slurp/cat (email@example.com;http://www.inktomi.com/slurp.html)¦18.104.22.168
2) Sat Sep 3 18:46:29 2000 --¦Slurp/cat (firstname.lastname@example.org;http://www.inktomi.com/slurp.html)¦22.214.171.124
30 Sat Sep 4 18:46:44 2000 --¦Slurp/cat (email@example.com;http://www.inktomi.com/slurp.html)¦126.96.36.199
My point here is notice the time and date, what is the purpose of spidering (same page all three times btw) almost exactly 24 hours apart?