Welcome to WebmasterWorld Guest from 54.159.179.132

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

AllTheWeb

   
2:57 am on Feb 23, 2000 (gmt 0)

10+ Year Member



Hi ... any discussion on how AllTheWeb works ? Since they have the largest index now.

Wonder how they review the submission ? Using Spider or Human Review ?

If Spider, do they have an IP ?

Thanks for sharing !

8:54 am on Mar 3, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



They have a full roaming spider just like most of the other SE's. Theirs is a pretty thourgh spider that picks up almost everything. Submissions are not reviewed.
10:53 am on Mar 3, 2000 (gmt 0)

10+ Year Member



Wow ! Need to spend a lot of time tracking the spider.

Do you think Gateway page will work for this Search Engine ?

11:53 am on Mar 3, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Yes, gateway pages can work, but since you can only submit your root url, you need to make sure you "direct" the spider to pages you want it to see - either by some cgi process or by including hidden links on your your homepage.
12:43 am on Mar 4, 2000 (gmt 0)

10+ Year Member



Here are some IP's for Alltheweb (FAST)

User Agent: FAST (not the full user agent, but it will do for an inexact match)

209.67.247.15
209.67.244.78
209.67.244.79
209.67.247.153
209.67.247.156
209.67.247.201
209.67.247.207

As Brett said they gobble up everything, so make sure you have links to all the pages you want them to grab.

3:44 am on Mar 4, 2000 (gmt 0)

10+ Year Member



Air_: Possibly detected a typo or two in your list:

209.67.244.78
won't resolve under nslookup
rwhois: Edu.com

209.67.244.79
won't resolve under nslookup
rwhois: Edu.com

209.67.247.15
won't resolve under nslookup
rwhois: Worldstreet

You sure you weren't going to write "247" for the first two?
(Though it beats me where that Worldstreet IP would fit in then.)
So our spiderScouts dept. may be wrong in which case we would be
glad to stand corrected: could you, if so, include some log file
data, please?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month