bizarre robots.txt - Sitemaps, Meta Data, and robots.txt forum at WebmasterWorld - WebmasterWorld

Forum Moderators: goodroi

Message Too Old, No Replies

bizarre robots.txt

must be an old WebmasterWorld suggestion for robots.txt

AussieWebmaster

8:33 pm on Jul 20, 2016 (gmt 0)

10+ Year Member

[ameristarjet.com...]

they quote the list coming from WebmasterWorld

tangor

8:47 pm on Jul 20, 2016 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Top Contributors Of The Month

Those were the days ...

keyplyr

11:48 pm on Jul 20, 2016 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Top Contributors Of The Month

3/4's of those agents listed would not follow robots.txt no matter how old or who complied it. It's mostly bogus.