AlltheWeb and Altavista Merging (robots.txt atleast!)

As we speak.

[xcandyman]

Things are changing with AlltheWeb and Altavista I have got a feeling that will the same entity very soon. Have a look at each robots.txt and see what I mean. A babelfish mention is even in alltheweb robots.txt now!

ALLTHEWEB

# Tells Scanning Robots Where They Are And Are Not Welcome
#
# User-agent:can also specify by name; "*" is for everyone
# Disallow:disallow if this matches first part of requested path
#
# For now disallow all we can modify this as needed to allow certain crawlers.
#

User-agent: *

Disallow: /search
Disallow: /sidebar
Disallow: /advanced
Disallow: /alchemist
Disallow: /customize
Disallow: /go
Disallow: /go2
Disallow: /cgi-bin
Disallow: /g/
Disallow: /web
Disallow: /r
Disallow: /babelfish
Disallow: /urltrurl
Disallow: /translate
Disallow: /image/results
Disallow: /image/samepage
Disallow: /image/res_detail
Disallow: /audio/results
Disallow: /audio/samepage
Disallow: /audio/res_detail
Disallow: /video/results
Disallow: /video/samepage
Disallow: /video/res_detail
Disallow: /news/more

ALTAVISTA

# Tells Scanning Robots Where They Are And Are Not Welcome
#
# User-agent:can also specify by name; "*" is for everyone
# Disallow:disallow if this matches first part of requested path
#
# For now disallow all we can modify this as needed to allow certain crawlers.
#

User-agent: *
Disallow: /cgi-bin
Disallow: /web
Disallow: /r
Disallow: /babelfish
Disallow: /urltrurl
Disallow: /translate
Disallow: /image/results
Disallow: /image/samepage
Disallow: /image/res_detail
Disallow: /audio/results
Disallow: /audio/samepage
Disallow: /audio/res_detail
Disallow: /video/results
Disallow: /video/samepage
Disallow: /video/res_detail
Disallow: /news/more

See the simularites?

[heini]

Good find, thanks. As one of the engineers pointed out both engines are in the process of getting integrated on the frontend.
The PFI programs are expected to be merged until mid October.
The serps however still are seperate.

[dougmcc1]

Wasn't the reason that AllTheWeb claimed to have the largest index a few weeks ago due to a merging of their indexes?

[heini]

I don't think so, indexes are very different.

[dougmcc1]

Thanks heini. After doing a simple search it does seem that ATW isn't using AV's results. The search I tried returns a result in AV which isn't returned in ATW.

Either way, according to [searchengine-news.com...] :

FAST Add URL now emulates AltaVista's

http*//addurl.alltheweb.com/add_url
http*//addurl.altavista.com/addurl/new

[heini]

Yes, the frontend for both engines is one and the same, however options, output and backend are different:
[webmasterworld.com...]

>Addurl:
Only makes sense, as basically the PFI options will be promoted through those channels.
Addurls have have for AV and ATW for a long time been the same as with Google: 99% webmaster placebo. They add pages for free depending on links.