homepage Welcome to WebmasterWorld Guest from 50.16.112.199
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
php and 'include' files
charlie12345

10+ Year Member



 
Msg#: 312 posted 12:45 pm on Mar 6, 2004 (gmt 0)

My site consists of php files, which 'include' various '.inc' files. Should I list all the '.inc' files as DISALLOW? Or would that stop the robot/spider from following the content of the .php pages....

Thanks,
Charlie

 

lorax

WebmasterWorld Administrator lorax us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 312 posted 2:12 am on Mar 11, 2004 (gmt 0)

Welcome to WebmasterWorld charlie12345,
Don't even mention those files in your robots.txt. The bots will never know those files exist if you don't mention them. Hackers will know they exist if you stick them in your robots.txt file!

scottiecla

10+ Year Member



 
Msg#: 312 posted 3:19 am on Mar 11, 2004 (gmt 0)

Hi Charlie-

Since you don't link directly to the included files (they are already compiled and returned as a complete page to the browser which is what the spider will get) there's no reason the spider would find them and crawl them separately.

So, no, you don't need to exclude them. :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved