Welcome to WebmasterWorld Guest from 35.175.120.174

Forum Moderators: goodroi

Message Too Old, No Replies

bizarre robots.txt

must be an old WebmasterWorld suggestion for robots.txt

     
8:33 pm on Jul 20, 2016 (gmt 0)

Full Member

10+ Year Member

joined:July 29, 2003
posts: 256
votes: 2


[ameristarjet.com...]

they quote the list coming from WebmasterWorld
8:47 pm on July 20, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10599
votes: 1128


Those were the days ...
11:48 pm on July 20, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


3/4's of those agents listed would not follow robots.txt no matter how old or who complied it. It's mostly bogus.