Welcome to WebmasterWorld Guest from 107.20.75.63

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

AllTheWeb

     
2:57 am on Feb 23, 2000 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 26, 2000
posts:414
votes: 0


Hi ... any discussion on how AllTheWeb works ? Since they have the largest index now.

Wonder how they review the submission ? Using Spider or Human Review ?

If Spider, do they have an IP ?

Thanks for sharing !

8:54 am on Mar 3, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


They have a full roaming spider just like most of the other SE's. Theirs is a pretty thourgh spider that picks up almost everything. Submissions are not reviewed.
10:53 am on Mar 3, 2000 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 26, 2000
posts:414
votes: 0


Wow ! Need to spend a lot of time tracking the spider.

Do you think Gateway page will work for this Search Engine ?

11:53 am on Mar 3, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


Yes, gateway pages can work, but since you can only submit your root url, you need to make sure you "direct" the spider to pages you want it to see - either by some cgi process or by including hidden links on your your homepage.

Air_

12:43 am on Mar 4, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


Here are some IP's for Alltheweb (FAST)

User Agent: FAST (not the full user agent, but it will do for an inexact match)

209.67.247.15
209.67.244.78
209.67.244.79
209.67.247.153
209.67.247.156
209.67.247.201
209.67.247.207

As Brett said they gobble up everything, so make sure you have links to all the pages you want them to grab.

3:44 am on Mar 4, 2000 (gmt 0)

Full Member

10+ Year Member

joined:Sept 23, 1999
posts:201
votes: 0


Air_: Possibly detected a typo or two in your list:

209.67.244.78
won't resolve under nslookup
rwhois: Edu.com

209.67.244.79
won't resolve under nslookup
rwhois: Edu.com

209.67.247.15
won't resolve under nslookup
rwhois: Worldstreet

You sure you weren't going to write "247" for the first two?
(Though it beats me where that Worldstreet IP would fit in then.)
So our spiderScouts dept. may be wrong in which case we would be
glad to stand corrected: could you, if so, include some log file
data, please?