Welcome to WebmasterWorld Guest from 54.159.165.175

Forum Moderators: goodroi

How to stop robots on .php files

   
1:16 pm on Feb 10, 2004 (gmt 0)

10+ Year Member



I have a site and I want to stop google and all other robots finding the .php pages. All these pages have extended dynamic URLS.

Is there any way I can shorthand the robots.txt to stop all these pages being indexed.

Maybe:

User-agent: googlebot

Disallow: /*.php

cheers
Dave

6:20 pm on Feb 10, 2004 (gmt 0)

WebmasterWorld Administrator bakedjake is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Is there any way I can shorthand the robots.txt to stop all these pages being indexed.

Short of putting them in a directory called /php/ and excluding that directory, no.

6:40 pm on Feb 10, 2004 (gmt 0)

10+ Year Member



GoogleBot will support doing what you want. Read their webmaster faq for details and an example (which is pretty similiar to what you did).

However, other bots will not support this at all. There is no way to test such a non-standard robots.txt file. It is best to put your php files under a seperate directory (as bakedjake said). If you absolutely can't, you should at least use the ROBOTS NOINDEX meta tag, and also be sure PHP warning messages are turned off.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month