AllTheWeb

[David]

Hi ... any discussion on how AllTheWeb works ? Since they have the largest index now.

Wonder how they review the submission ? Using Spider or Human Review ?

If Spider, do they have an IP ?

Thanks for sharing !

[Brett_Tabke]

They have a full roaming spider just like most of the other SE's. Theirs is a pretty thourgh spider that picks up almost everything. Submissions are not reviewed.

[David]

Wow ! Need to spend a lot of time tracking the spider.

Do you think Gateway page will work for this Search Engine ?

[Brett_Tabke]

Yes, gateway pages can work, but since you can only submit your root url, you need to make sure you "direct" the spider to pages you want it to see - either by some cgi process or by including hidden links on your your homepage.

[Air_]

Here are some IP's for Alltheweb (FAST)

User Agent: FAST (not the full user agent, but it will do for an inexact match)

209.67.247.15
209.67.244.78
209.67.244.79
209.67.247.153
209.67.247.156
209.67.247.201
209.67.247.207

As Brett said they gobble up everything, so make sure you have links to all the pages you want them to grab.

[fantomaster]

Air_: Possibly detected a typo or two in your list:

209.67.244.78
won't resolve under nslookup
rwhois: Edu.com

209.67.244.79
won't resolve under nslookup
rwhois: Edu.com

209.67.247.15
won't resolve under nslookup
rwhois: Worldstreet

You sure you weren't going to write "247" for the first two?
(Though it beats me where that Worldstreet IP would fit in then.)
So our spiderScouts dept. may be wrong in which case we would be
glad to stand corrected: could you, if so, include some log file
data, please?