2by4 - 4:18 am on Nov 30, 2005 (gmt 0)

"I'll bet *anything* that with a Googlebot IP address and useragent you again would not see all useragents dissallowed in robots.txt, and could crawl the site. But that I guess we'll never know..."

I'll bet you anything that at the time of this posting robots.txt is being cloaked by useragent.

I was just regretting not grabbing the old robots.txt for its bot list, but there it is, just like it used to be, no search bots banned at all.... now it's saved though... brett is still one of my favorite tricksters out there, I learn a lot from him, and some others here, all the time.

by the way, how do you cloak a .txt file? I remember trying that with .css but I couldn't figure it out, didn't spend very long on it though, that's in an apache php environment, is it just a htaccess thing? Anybody feel like sharing the code?

I've never seen the point of these though:

User-agent: EmailSiphon
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: EmailWolf
Disallow: /

do these actually obey robots.txt? Somehow I doubt it.

Of course the real question is what a google ip is seeing.

to the poster asking why brett doesn't just add hardware, he's answered that several times, the bbs software isn't written for multiple servers yet, although he's working on it.