using robots.txt

is it necessary to exclude .php pages

[deborahbaker]

Can anyone tell me if it is necessary or better to disallow .php files in the robots.txt file. I am concerned that the .php files in my site are affecting its ranking with the search engines? Will I get a better ranking if I disallow these files? Thanks

[mattur]

No, it's not better or neccessary to disallow php pages in your robots.txt - especially if you want those php pages to appear in the search engines :)

Perhaps if you explain why you think your php pages are having a negative ranking effect someone here may be able to advise you further.

[g1smd]

The output of your PHP script is a page of valid HTML code. Spiders cannot see any difference in that HTML code than what they can see in the HTML code on a page that has a filename eding in .html or .anything else.

[deborahbaker]

Ok thanks very much both for that info. I am just trying to understand how spiders work as I am having a problem getting my .php sites high with Google and I am using a process of elimination to try and isolate the problem. I also thought I read that spiders dont like .php files and that some engines cant read them.

Am I better to include keywords in my .php files too then? Will they get read?

[mattur]

Do a view source from your browser on one of your php pages. That is what the spiders see. It's a html web page, exactly like any other web page, so can be optimised exactly like a static html page.

PHP runs on the server to generate the html code, and you have complete control of the html in exactly the same way as when you write a static html page in your editor.

The only potential problem I can think of is where php is set to append a sessionid onto the end of your urls. If you're not doing this, spiders/search engines will have no problems indexing your php pages.

[markgibson77]

Can anyone tell me if its better to disallow my .php include files from search engines spiders? I have been told that Google does not like seeing two pages with exactly the same instances. Is this correct?