homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

using robots.txt
is it necessary to exclude .php pages

 2:13 pm on Jun 26, 2006 (gmt 0)

Can anyone tell me if it is necessary or better to disallow .php files in the robots.txt file. I am concerned that the .php files in my site are affecting its ranking with the search engines? Will I get a better ranking if I disallow these files? Thanks



 12:48 pm on Jun 27, 2006 (gmt 0)

No, it's not better or neccessary to disallow php pages in your robots.txt - especially if you want those php pages to appear in the search engines :)

Perhaps if you explain why you think your php pages are having a negative ranking effect someone here may be able to advise you further.


 8:56 pm on Jul 1, 2006 (gmt 0)

The output of your PHP script is a page of valid HTML code. Spiders cannot see any difference in that HTML code than what they can see in the HTML code on a page that has a filename eding in .html or .anything else.


 10:23 am on Jul 4, 2006 (gmt 0)

Ok thanks very much both for that info. I am just trying to understand how spiders work as I am having a problem getting my .php sites high with Google and I am using a process of elimination to try and isolate the problem. I also thought I read that spiders dont like .php files and that some engines cant read them.

Am I better to include keywords in my .php files too then? Will they get read?


 11:00 am on Jul 4, 2006 (gmt 0)

Do a view source from your browser on one of your php pages. That is what the spiders see. It's a html web page, exactly like any other web page, so can be optimised exactly like a static html page.

PHP runs on the server to generate the html code, and you have complete control of the html in exactly the same way as when you write a static html page in your editor.

The only potential problem I can think of is where php is set to append a sessionid onto the end of your urls. If you're not doing this, spiders/search engines will have no problems indexing your php pages.


 11:53 am on Jul 13, 2006 (gmt 0)

Can anyone tell me if its better to disallow my .php include files from search engines spiders? I have been told that Google does not like seeing two pages with exactly the same instances. Is this correct?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved