Welcome to WebmasterWorld Guest from 54.196.214.35

Forum Moderators: incrediBILL & lawman

Message Too Old, No Replies

Brett's new Bot Blog

Check it out...

     
6:09 pm on Dec 20, 2005 (gmt 0)

New User

10+ Year Member

joined:Oct 25, 2005
posts:12
votes: 0


I was curious to see how Brett's "bot war" was progressing, so I took a quick look at his robots.txt file.

Interesting reading...

Thanks, Brett!

6:28 pm on Dec 20, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Oct 26, 2002
posts:3292
votes: 6


he, he, he.

That's great. Only an old BBS hand would come up with something like that.

7:12 pm on Dec 20, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


That's interesting - I had no idea Brett coined the term "SERP".
7:17 pm on Dec 20, 2005 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12166
votes: 51


That's a great idea!

I didn't know he coined that phrase either.

Has Brett started a new craze? Are we going to see robots.txt files become the next marketing medium? ;)

7:24 pm on Dec 20, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 28, 2005
posts:614
votes: 0


That's the best robots.txt I've ever read.

Freq---

10:10 pm on Dec 20, 2005 (gmt 0)

Senior Member from FR 

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Feb 15, 2004
posts:6717
votes: 230


pageone' ..do not give people ideas ..all we need is keyword spamming of robots text ..this is just Mr Tabke showing his underworld credentials ..next he'll get into graphics in the robots text and regfiles ..;)
12:13 am on Dec 21, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 12, 2005
posts:95
votes: 0


We need some good ASCII art in there, Brett: maybe Santa and his sleigh, just to complete the retro feel.

(Wonders how to send a trackback from his own robots.txt...)

12:17 am on Dec 21, 2005 (gmt 0)

Senior Member from MY 

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 1, 2003
posts:4847
votes: 0


That's a really imaginative idea. I'm wondering if there is a market for a cgi to manage the posting / edits / etc. for the robots.txt - may even manage comments via external links?

I'm almost inspired to write a whole article and put it in an html comment tag now ;-)

I'd be interested to see what would happen if someone put a link to the robots.txt file. Whilst we know robots.txt is indexed - would Google dutifully ignore the commented lines, or would it happy serve it within actual SERPS? If only I knew someone with a website :-(

12:30 am on Dec 21, 2005 (gmt 0)

Senior Member from FR 

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Feb 15, 2004
posts:6717
votes: 230


I can just see it ..Brett adds "greetz fly out" and the "we may be on IRC but maybe not" to the robots text ..

retro with chrome fenders and old school flames ..

so who's first to get a robots.txt to top of serps in G then ..? ;)

TFMC_Mark

12:53 am on Dec 21, 2005 (gmt 0)

Inactive Member
Account Expired

 
 


For some reason I have found myself blogging about webmasterworld twice in the past couple of months and both times it has been because I have been amazed by the robots.txt.

When all bots were banned I raved, and now there is a robots.txt bot blog. Brett is trying to be old_school cool and it is working for me. (so much so that I come out of lurking mode for a brief post.)

10:13 pm on Dec 21, 2005 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


Thanks Mark.
4:21 am on Dec 22, 2005 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 20, 2003
posts:408
votes: 0


WebmasterWorld - Expect the unexpected!

Some ascii art would give it a little boost.

Talikng about the retro feel - I got shudders thinking about the LARGE letters you used to get on the first page of green bar report printouts like:

VM/CMS or IBM/3270

But I digress...

8:02 am on Dec 22, 2005 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 26, 2004
posts:3145
votes: 12


IoI I *love* finding things on my own that I would only be able to find being a web designer (though I didn't see that until I read the thread). A good hidden read. It's an area of interest and I'd like to learn from Brett's experiences and his observations as well. I'd have to actually stop and think if I was offered a choice between a room full of gold or an access log from WW, Wiki, Google, or Yahoo!

I would like to add that the AdWords trick people most likely (in my kind of humble opinion heh) that most people's screens are not optimized. Many people EEK when I max out (hopefully 100% unless there is tearing then typically between 84-92% on cheapo-screens) the RGB setting without realizing that it, brightness, and contrast are three very separate things. Honestly the only reason I know the difference is because I tweak everything I can that I understand including my hardware. So long as monitors are sent unoptimized to non-tech-savvy computer users this screen contrast trick should continue to work I suspect.

11:13 am on Dec 22, 2005 (gmt 0)

Senior Member from MY 

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 1, 2003
posts:4847
votes: 0


AdWords trick people most likely (in my kind of humble opinion heh) that most people's screens are not optimized

Not sure I follow?

11:22 am on Dec 22, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 11, 2002
posts:2024
votes: 0


@pageoneresults:
Are we going to see robots.txt files become the next marketing medium?

Reading the

robots.txt
of competitive sites is a must for me!

@Leosghost:
Have you tried [google.com...] - seems Brett is in a good neighbourhood.

11:33 am on Dec 22, 2005 (gmt 0)

Senior Member from MY 

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 1, 2003
posts:4847
votes: 0


Are we going to see robots.txt files become the next marketing medium?

A firefox plugin to show the robots.txt in a window / popup anyone?

3:56 pm on Dec 22, 2005 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member jab_creations is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 26, 2004
posts:3145
votes: 12


Vince, most people's screens have low RGB settings, and various levels of brightness and contrast (the physical controls/options besides the on/off button for clarification). So if you put my screen next to someone else's you definitely see things on my screen (optimized) that a non-savvy user would not be able to see. The same if you see a rack of screens all with various settings in a store and again are unoptimized at least in some cases I'd imagine in that sort of scenario.
10:30 am on Dec 23, 2005 (gmt 0)

Senior Member from FR 

WebmasterWorld Senior Member leosghost is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Feb 15, 2004
posts:6717
votes: 230


@Brett.. I just noticed something , should n't
# [webmasterworld.com...]
# Actual robots can always be found here for: [webmasterworld.com...]
# Old full robots.txt can be found here: [webmasterworld.com...]
read
# [webmasterworld.com...]
# Actual robots .txt can always be found here for: [webmasterworld.com...]
# Old full robots.txt can be found here: [webmasterworld.com...]

or are you running an aff deal via the backdoor with 20th century fox for the movie toys? ;)

2:29 pm on Dec 28, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 11, 2002
posts:2024
votes: 0


Getting a 404 right now.
10:59 pm on Dec 27, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


What happened to it? Hopefully, it's coming back. I really enjoy reading it. :)
1:33 am on Dec 28, 2005 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 25, 2003
posts:891
votes: 59


Unlike 99.99% of all bloggers who publish for a week and then get on with their lives leaving a tiny bit of their life littering the web Brett has picked up his comments.

I also thought it great fun...

Save the WebmasterWorld robots.txt Blog

New WWW Life Form Missing

I Scream, You Scream, We all Scream for robots.txt Blog

2:40 pm on Dec 28, 2005 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


ahhh, the dangers of a scripted bots.txt. ;-)

...should work now (tm)

10:02 pm on Jan 2, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member whoisgregg is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 9, 2003
posts:3416
votes: 0


So, most blogs I just subscribe to the RSS feed. I've already checked for a /robots.rss or /robots.xml with no luck. :( So, this blog I have to remember to check every day.

Is it ironic that this is the first time I've felt like programming a robot for my personal use? :)