Google+Include pages

[stevelibby]

Hi all
i need some advise as i am now not sure what to do.
I changed my web site last year to using asp code from static html. Everything is working fine, however i have noticed that google has indexed all of my include files when it was asked not to via robots.txt?
What can i do?
Also on the subject of include pages should you leave the html/head comments in or not?
I was not sure so i took them out and left the table tags in, is this what you asp people do?

[TheNige]

How was it able to spider your includes? I think you'd have to allow directory browsing on your site or have actual links to your includes visible to the spiders.

[stevelibby]

This is the part i am unable to work out, i wrote it in the robots.txt not to go there and yet it has!
What can i do? I am currently moving the includes to a diffrent directory and then once done i shall open up the folder for google to go at again, however this time i will have empty pages, i can think of any other way, can you?

[mattglet]

How do the spiders even know about your includes directories? Includes are processed on the server, so any references to them should never been seen on the client.

[TheNige]

exactly...Don't even put it in your robots.txt...because they shouldn't even find them because spiders follow links. And the path to the include is not available to the client.

Is the site set to allow directory browsing? That is the only way I can think that they could find it.

[stevelibby]

this is the part that has confused me, there is no links in my site to those pages?
What if a web site found those links and and placed them on their web site for the engines to follows through, would that then alter my rankings?
Now that they have been indexed i have now moved the includes pages elsewhere, and what i am going to do is clean all pages to nothing and then let engine go through and remove its cached. Would you do this?

[devil_dog]

the best solution....

1) place your includes in a seperate directory.
2) allow only localhost access to this directory...