Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

php and 'include' files



12:45 pm on Mar 6, 2004 (gmt 0)

Inactive Member
Account Expired


My site consists of php files, which 'include' various '.inc' files. Should I list all the '.inc' files as DISALLOW? Or would that stop the robot/spider from following the content of the .php pages....


2:12 am on Mar 11, 2004 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lorax is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 31, 2002
votes: 0

Welcome to WebmasterWorld charlie12345,
Don't even mention those files in your robots.txt. The bots will never know those files exist if you don't mention them. Hackers will know they exist if you stick them in your robots.txt file!
3:19 am on Mar 11, 2004 (gmt 0)

New User

10+ Year Member

joined:Apr 27, 2003
votes: 0

Hi Charlie-

Since you don't link directly to the included files (they are already compiled and returned as a complete page to the browser which is what the spider will get) there's no reason the spider would find them and crawl them separately.

So, no, you don't need to exclude them. :)