homepage Welcome to WebmasterWorld Guest from 54.198.148.191
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to stop robots on .php files
djtaverner




msg:1526886
 1:16 pm on Feb 10, 2004 (gmt 0)

I have a site and I want to stop google and all other robots finding the .php pages. All these pages have extended dynamic URLS.

Is there any way I can shorthand the robots.txt to stop all these pages being indexed.

Maybe:

User-agent: googlebot

Disallow: /*.php

cheers
Dave

 

bakedjake




msg:1526887
 6:20 pm on Feb 10, 2004 (gmt 0)

Is there any way I can shorthand the robots.txt to stop all these pages being indexed.

Short of putting them in a directory called /php/ and excluding that directory, no.

Robert Thivierge




msg:1526888
 6:40 pm on Feb 10, 2004 (gmt 0)

GoogleBot will support doing what you want. Read their webmaster faq for details and an example (which is pretty similiar to what you did).

However, other bots will not support this at all. There is no way to test such a non-standard robots.txt file. It is best to put your php files under a seperate directory (as bakedjake said). If you absolutely can't, you should at least use the ROBOTS NOINDEX meta tag, and also be sure PHP warning messages are turned off.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved