homepage Welcome to WebmasterWorld Guest from 54.166.110.222
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
AlltheWeb and Altavista Merging (robots.txt atleast!)
As we speak.
xcandyman




msg:224567
 10:37 am on Oct 6, 2003 (gmt 0)

Things are changing with AlltheWeb and Altavista I have got a feeling that will the same entity very soon. Have a look at each robots.txt and see what I mean. A babelfish mention is even in alltheweb robots.txt now!

ALLTHEWEB
# Tells Scanning Robots Where They Are And Are Not Welcome
#
# User-agent:can also specify by name; "*" is for everyone
# Disallow:disallow if this matches first part of requested path
#
# For now disallow all we can modify this as needed to allow certain crawlers.
#

User-agent: *

Disallow: /search
Disallow: /sidebar
Disallow: /advanced
Disallow: /alchemist
Disallow: /customize
Disallow: /go
Disallow: /go2
Disallow: /cgi-bin
Disallow: /g/
Disallow: /web
Disallow: /r
Disallow: /babelfish
Disallow: /urltrurl
Disallow: /translate
Disallow: /image/results
Disallow: /image/samepage
Disallow: /image/res_detail
Disallow: /audio/results
Disallow: /audio/samepage
Disallow: /audio/res_detail
Disallow: /video/results
Disallow: /video/samepage
Disallow: /video/res_detail
Disallow: /news/more

ALTAVISTA
# Tells Scanning Robots Where They Are And Are Not Welcome
#
# User-agent:can also specify by name; "*" is for everyone
# Disallow:disallow if this matches first part of requested path
#
# For now disallow all we can modify this as needed to allow certain crawlers.
#

User-agent: *
Disallow: /cgi-bin
Disallow: /web
Disallow: /r
Disallow: /babelfish
Disallow: /urltrurl
Disallow: /translate
Disallow: /image/results
Disallow: /image/samepage
Disallow: /image/res_detail
Disallow: /audio/results
Disallow: /audio/samepage
Disallow: /audio/res_detail
Disallow: /video/results
Disallow: /video/samepage
Disallow: /video/res_detail
Disallow: /news/more

See the simularites?

 

heini




msg:224568
 10:53 am on Oct 6, 2003 (gmt 0)

Good find, thanks. As one of the engineers pointed out both engines are in the process of getting integrated on the frontend.
The PFI programs are expected to be merged until mid October.
The serps however still are seperate.

dougmcc1




msg:224569
 2:20 pm on Oct 6, 2003 (gmt 0)

Wasn't the reason that AllTheWeb claimed to have the largest index a few weeks ago due to a merging of their indexes?

heini




msg:224570
 2:24 pm on Oct 6, 2003 (gmt 0)

I don't think so, indexes are very different.

dougmcc1




msg:224571
 7:04 pm on Oct 6, 2003 (gmt 0)

Thanks heini. After doing a simple search it does seem that ATW isn't using AV's results. The search I tried returns a result in AV which isn't returned in ATW.

Either way, according to [searchengine-news.com...] :
FAST Add URL now emulates AltaVista's

http*//addurl.alltheweb.com/add_url
http*//addurl.altavista.com/addurl/new

heini




msg:224572
 7:49 pm on Oct 6, 2003 (gmt 0)

Yes, the frontend for both engines is one and the same, however options, output and backend are different:
[webmasterworld.com...]

>Addurl:
Only makes sense, as basically the PFI options will be promoted through those channels.
Addurls have have for AV and ATW for a long time been the same as with Google: 99% webmaster placebo. They add pages for free depending on links.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved