Welcome to WebmasterWorld Guest from 54.166.55.74

Forum Moderators: open

Message Too Old, No Replies

AlltheWeb and Altavista Merging (robots.txt atleast!)

As we speak.

   
10:37 am on Oct 6, 2003 (gmt 0)

10+ Year Member



Things are changing with AlltheWeb and Altavista I have got a feeling that will the same entity very soon. Have a look at each robots.txt and see what I mean. A babelfish mention is even in alltheweb robots.txt now!

ALLTHEWEB

# Tells Scanning Robots Where They Are And Are Not Welcome
#
# User-agent:can also specify by name; "*" is for everyone
# Disallow:disallow if this matches first part of requested path
#
# For now disallow all we can modify this as needed to allow certain crawlers.
#

User-agent: *

Disallow: /search
Disallow: /sidebar
Disallow: /advanced
Disallow: /alchemist
Disallow: /customize
Disallow: /go
Disallow: /go2
Disallow: /cgi-bin
Disallow: /g/
Disallow: /web
Disallow: /r
Disallow: /babelfish
Disallow: /urltrurl
Disallow: /translate
Disallow: /image/results
Disallow: /image/samepage
Disallow: /image/res_detail
Disallow: /audio/results
Disallow: /audio/samepage
Disallow: /audio/res_detail
Disallow: /video/results
Disallow: /video/samepage
Disallow: /video/res_detail
Disallow: /news/more

ALTAVISTA

# Tells Scanning Robots Where They Are And Are Not Welcome
#
# User-agent:can also specify by name; "*" is for everyone
# Disallow:disallow if this matches first part of requested path
#
# For now disallow all we can modify this as needed to allow certain crawlers.
#

User-agent: *
Disallow: /cgi-bin
Disallow: /web
Disallow: /r
Disallow: /babelfish
Disallow: /urltrurl
Disallow: /translate
Disallow: /image/results
Disallow: /image/samepage
Disallow: /image/res_detail
Disallow: /audio/results
Disallow: /audio/samepage
Disallow: /audio/res_detail
Disallow: /video/results
Disallow: /video/samepage
Disallow: /video/res_detail
Disallow: /news/more

See the simularites?

10:53 am on Oct 6, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Good find, thanks. As one of the engineers pointed out both engines are in the process of getting integrated on the frontend.
The PFI programs are expected to be merged until mid October.
The serps however still are seperate.
2:20 pm on Oct 6, 2003 (gmt 0)

10+ Year Member



Wasn't the reason that AllTheWeb claimed to have the largest index a few weeks ago due to a merging of their indexes?
2:24 pm on Oct 6, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I don't think so, indexes are very different.
7:04 pm on Oct 6, 2003 (gmt 0)

10+ Year Member



Thanks heini. After doing a simple search it does seem that ATW isn't using AV's results. The search I tried returns a result in AV which isn't returned in ATW.

Either way, according to [searchengine-news.com...] :

FAST Add URL now emulates AltaVista's

http*//addurl.alltheweb.com/add_url
http*//addurl.altavista.com/addurl/new
7:49 pm on Oct 6, 2003 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Yes, the frontend for both engines is one and the same, however options, output and backend are different:
[webmasterworld.com...]

>Addurl:
Only makes sense, as basically the PFI options will be promoted through those channels.
Addurls have have for AV and ATW for a long time been the same as with Google: 99% webmaster placebo. They add pages for free depending on links.