xcandyman - 10:37 am on Oct 6, 2003 (gmt 0)

Things are changing with AlltheWeb and Altavista I have got a feeling that will the same entity very soon. Have a look at each robots.txt and see what I mean. A babelfish mention is even in alltheweb robots.txt now!

ALLTHEWEB

# Tells Scanning Robots Where They Are And Are Not Welcome # # User-agent:can also specify by name; "*" is for everyone # Disallow:disallow if this matches first part of requested path # # For now disallow all we can modify this as needed to allow certain crawlers. # User-agent: * Disallow: /search Disallow: /sidebar Disallow: /advanced Disallow: /alchemist Disallow: /customize Disallow: /go Disallow: /go2 Disallow: /cgi-bin Disallow: /g/ Disallow: /web Disallow: /r Disallow: /babelfish Disallow: /urltrurl Disallow: /translate Disallow: /image/results Disallow: /image/samepage Disallow: /image/res_detail Disallow: /audio/results Disallow: /audio/samepage Disallow: /audio/res_detail Disallow: /video/results Disallow: /video/samepage Disallow: /video/res_detail Disallow: /news/more

ALTAVISTA

# Tells Scanning Robots Where They Are And Are Not Welcome # # User-agent:can also specify by name; "*" is for everyone # Disallow:disallow if this matches first part of requested path # # For now disallow all we can modify this as needed to allow certain crawlers. # User-agent: * Disallow: /cgi-bin Disallow: /web Disallow: /r Disallow: /babelfish Disallow: /urltrurl Disallow: /translate Disallow: /image/results Disallow: /image/samepage Disallow: /image/res_detail Disallow: /audio/results Disallow: /audio/samepage Disallow: /audio/res_detail Disallow: /video/results Disallow: /video/samepage Disallow: /video/res_detail Disallow: /news/more

See the simularites?