Can anyone tell me if it is necessary or better to disallow .php files in the robots.txt file. I am concerned that the .php files in my site are affecting its ranking with the search engines? Will I get a better ranking if I disallow these files? Thanks
The output of your PHP script is a page of valid HTML code. Spiders cannot see any difference in that HTML code than what they can see in the HTML code on a page that has a filename eding in .html or .anything else.
Ok thanks very much both for that info. I am just trying to understand how spiders work as I am having a problem getting my .php sites high with Google and I am using a process of elimination to try and isolate the problem. I also thought I read that spiders dont like .php files and that some engines cant read them.
Am I better to include keywords in my .php files too then? Will they get read?
Do a view source from your browser on one of your php pages. That is what the spiders see. It's a html web page, exactly like any other web page, so can be optimised exactly like a static html page.
PHP runs on the server to generate the html code, and you have complete control of the html in exactly the same way as when you write a static html page in your editor.
The only potential problem I can think of is where php is set to append a sessionid onto the end of your urls. If you're not doing this, spiders/search engines will have no problems indexing your php pages.
Can anyone tell me if its better to disallow my .php include files from search engines spiders? I have been told that Google does not like seeing two pages with exactly the same instances. Is this correct?