Welcome to WebmasterWorld Guest from 54.160.187.160

Forum Moderators: open

Message Too Old, No Replies

AlltheWeb and Altavista Merging (robots.txt atleast!)

As we speak.

     
10:37 am on Oct 6, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:June 20, 2002
posts:439
votes: 0


Things are changing with AlltheWeb and Altavista I have got a feeling that will the same entity very soon. Have a look at each robots.txt and see what I mean. A babelfish mention is even in alltheweb robots.txt now!

ALLTHEWEB

# Tells Scanning Robots Where They Are And Are Not Welcome
#
# User-agent:can also specify by name; "*" is for everyone
# Disallow:disallow if this matches first part of requested path
#
# For now disallow all we can modify this as needed to allow certain crawlers.
#

User-agent: *

Disallow: /search
Disallow: /sidebar
Disallow: /advanced
Disallow: /alchemist
Disallow: /customize
Disallow: /go
Disallow: /go2
Disallow: /cgi-bin
Disallow: /g/
Disallow: /web
Disallow: /r
Disallow: /babelfish
Disallow: /urltrurl
Disallow: /translate
Disallow: /image/results
Disallow: /image/samepage
Disallow: /image/res_detail
Disallow: /audio/results
Disallow: /audio/samepage
Disallow: /audio/res_detail
Disallow: /video/results
Disallow: /video/samepage
Disallow: /video/res_detail
Disallow: /news/more

ALTAVISTA

# Tells Scanning Robots Where They Are And Are Not Welcome
#
# User-agent:can also specify by name; "*" is for everyone
# Disallow:disallow if this matches first part of requested path
#
# For now disallow all we can modify this as needed to allow certain crawlers.
#

User-agent: *
Disallow: /cgi-bin
Disallow: /web
Disallow: /r
Disallow: /babelfish
Disallow: /urltrurl
Disallow: /translate
Disallow: /image/results
Disallow: /image/samepage
Disallow: /image/res_detail
Disallow: /audio/results
Disallow: /audio/samepage
Disallow: /audio/res_detail
Disallow: /video/results
Disallow: /video/samepage
Disallow: /video/res_detail
Disallow: /news/more

See the simularites?

10:53 am on Oct 6, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
posts:4404
votes: 0


Good find, thanks. As one of the engineers pointed out both engines are in the process of getting integrated on the frontend.
The PFI programs are expected to be merged until mid October.
The serps however still are seperate.
2:20 pm on Oct 6, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 28, 2003
posts:480
votes: 1


Wasn't the reason that AllTheWeb claimed to have the largest index a few weeks ago due to a merging of their indexes?
2:24 pm on Oct 6, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
posts:4404
votes: 0


I don't think so, indexes are very different.
7:04 pm on Oct 6, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Apr 28, 2003
posts:480
votes: 1


Thanks heini. After doing a simple search it does seem that ATW isn't using AV's results. The search I tried returns a result in AV which isn't returned in ATW.

Either way, according to [searchengine-news.com...] :

FAST Add URL now emulates AltaVista's

http*//addurl.alltheweb.com/add_url
http*//addurl.altavista.com/addurl/new
7:49 pm on Oct 6, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
posts:4404
votes: 0


Yes, the frontend for both engines is one and the same, however options, output and backend are different:
[webmasterworld.com...]

>Addurl:
Only makes sense, as basically the PFI options will be promoted through those channels.
Addurls have have for AV and ATW for a long time been the same as with Google: 99% webmaster placebo. They add pages for free depending on links.