homepage Welcome to WebmasterWorld Guest from 54.161.246.212
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
the wrong way to do robots.txt
kinda funny, whitehouse.gov robots file
amznVibe

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 192 posted 6:57 am on Oct 28, 2003 (gmt 0)

Not touching the political side of this, but check out the whitehouse.gov robots.txt file [whitehouse.gov]
Don't even know where to begin telling them how incorrectly that is done.
I think they need a WebmasterWorld membership ;)

 

onedumbear

10+ Year Member



 
Msg#: 192 posted 8:10 am on Oct 28, 2003 (gmt 0)

yup, that's funny alright.
whats got me curious is what were you doing looking at that? What motivated you to say, "hey, i wonder what the whitehouse robots.txt looks like"?
:)

amznVibe

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 192 posted 8:16 am on Oct 28, 2003 (gmt 0)

Actually I was wondering what CMS (content management) they used, and while Googling for it I found a blog comment about the robots.txt ;) so nothing evil I swear (hey what are those men in the dark suits doing at my door?)

graywolf

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 192 posted 12:39 pm on Oct 28, 2003 (gmt 0)

Do you think the dark suits look at the logs? If they do can you imagine them all sitting around wondering whats going on.

Sinner_G

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 192 posted 12:47 pm on Oct 28, 2003 (gmt 0)

All of you, please look right into the red light. What you saw was a digital weather balloon.

kevinpate

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 192 posted 12:53 pm on Oct 28, 2003 (gmt 0)

I will resist the obvious comment about the Disallow parameters being deliberately scooched way over to the right ... or maybe I won't, hehehehe

mole

10+ Year Member



 
Msg#: 192 posted 1:02 pm on Oct 28, 2003 (gmt 0)

I like the irony in this entry:
Disallow:/sitemap.html

RonPK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 192 posted 12:25 pm on Nov 1, 2003 (gmt 0)

Definitely the funniest robots.txt I've ever seen!

Disallow: /firstlady/recipes/iraq
;)

Daniel_O

5+ Year Member



 
Msg#: 192 posted 3:43 am on Nov 3, 2003 (gmt 0)

So many interesting URLs. However, every path I tried to explore turned out 404. Did anyone else experience this?

Daniel Odulo

wkitty42

10+ Year Member



 
Msg#: 192 posted 9:28 pm on Nov 8, 2003 (gmt 0)

whoa... hahahahahaha... seems that they are trying to prevent bots from accessing everything on the site... guess they don't know or understand that many bots can be configured to ignore robots.txt and to even present a real looking UA string... too funny... it appears that they update it regularly from looking at the section that contains the blocks for the new releases section... its got the current year and month...

Receptional

WebmasterWorld Administrator receptional us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 192 posted 7:31 pm on Nov 10, 2003 (gmt 0)

We've had clients who want to NOT get found on search engines.

It happens.

Of course, robots.txt is a voluntary protocol.

I note they are no.1 on Yahoo for "Whitehouse" - just above a sex site at whitehouse.com.

bcolflesh

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 192 posted 7:41 pm on Nov 10, 2003 (gmt 0)

Don't forget the .org hilarious parody site right behind them in the results.

Chndru

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 192 posted 7:42 pm on Nov 10, 2003 (gmt 0)

LOL.
Thanks bcolflesh ;)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved