homepage Welcome to WebmasterWorld Guest from 54.237.98.229
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to stop robots on .php files
djtaverner

10+ Year Member



 
Msg#: 277 posted 1:16 pm on Feb 10, 2004 (gmt 0)

I have a site and I want to stop google and all other robots finding the .php pages. All these pages have extended dynamic URLS.

Is there any way I can shorthand the robots.txt to stop all these pages being indexed.

Maybe:

User-agent: googlebot

Disallow: /*.php

cheers
Dave

 

bakedjake

WebmasterWorld Administrator bakedjake us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 277 posted 6:20 pm on Feb 10, 2004 (gmt 0)

Is there any way I can shorthand the robots.txt to stop all these pages being indexed.

Short of putting them in a directory called /php/ and excluding that directory, no.

Robert Thivierge

10+ Year Member



 
Msg#: 277 posted 6:40 pm on Feb 10, 2004 (gmt 0)

GoogleBot will support doing what you want. Read their webmaster faq for details and an example (which is pretty similiar to what you did).

However, other bots will not support this at all. There is no way to test such a non-standard robots.txt file. It is best to put your php files under a seperate directory (as bakedjake said). If you absolutely can't, you should at least use the ROBOTS NOINDEX meta tag, and also be sure PHP warning messages are turned off.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved