homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

Google pr0 despite robot.txt on site
Ndwbie needs advice with Google pr0

10+ Year Member

Msg#: 260 posted 2:32 am on Jan 15, 2003 (gmt 0)

I'm a newbie to this forum, and could not find this subject by search, so forgive me if it's been covered. My site has had robot.txt in the root saying "stay away" since day one, but suddenly my grey bar turned white - the dreaded pr0. My log files show googlebot stopping by once a month, but shows they didn't download any pages.

What did I do wrong, and how do I rescue my URL?



WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 260 posted 3:00 am on Jan 15, 2003 (gmt 0)

Welcome to the board, john3012.

First, off, having PR0 doesn't necessarily mean there's a penalty. Grey means the site is out of the index, all-white indicates that it's in, but there's not any PR assigned.

It's very easy to make syntax errors, especially common with naming the file. It's robots.txt (not robot.txt) so if you search on it plural you'll find plenty about it. Here's a tutorial for it, which is a good place to start:


If you want the site totally removed and not indexed, you can have it removed by Google immediately, following the instructions at their site. Otherwise, they'll pick it up if you get the robots.txt right.

If you want to be included, fix it, let them have it, and watch for the update after this coming one, which will be end of February. Just make sure there are enough quality inbound links to have PR in time for the crawl at the end of this month, without wasting any time - it's getting closer. Otherwise you still won't have PR showing, but not because of a penalty.

First thing is to get the file done and run it through the validator to double-check the syntax. A lot of people have problems because of it, and it's easily fixed.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved