another robots question!

[nixonuk]

Hi again

If pages are already listed in Google - will it make a difference if you had a robots.txt file?

I mean when those pages get crawled again will they then be binned from Google?

[jbinbpt]

Hi Nixonuk,
The robots.txt file is used to exclude portions of your site from being spidered or to exclude specific spiders from your site. So not having a robots.txt file is ok.

That said, We don't really know the specifics of ranking for any SE. Having a proper robots.txt file is good pratice and possibly encourages different spider activity. My opinion.

jb

[nixonuk]

Cheers.

What I meant really was that I have some pages that are indexed that I dont want indexed due to dupe content issues.

I therefore want to put a robots.txt file on those pages as I want them excluded from the search engines. So on Google's next crawl will it then bin those pages or is it already too late as they are already indexed?

Thanks

[jbinbpt]

I would put up the robots.txt file and allow the pages to fall out of the index without drawing additional attention to them. Don't worry too much about it. Micro-managing does not work.

jb

[specter]

Hi,

What I meant really was that I have some pages that are indexed that I dont want indexed due to dupe content issues.

I therefore want to put a robots.txt file on those pages as I want them excluded from the search engines. So on Google's next crawl will it then bin those pages or is it already too late as they are already indexed?

It's not too late.Even if they are already indexed.
Google refreshes the settings of each page every time it crawls.So,if you put a robt.txt file in them they would be excluded at the next crawling.