How to de index development version of my website?

[patel_rosh]

Hey all..:) I am in a spot of bother a bit.. Actually my website is in development mode but i forget to put robots.txt file in it so what happened that my website under developed pages have been crawled by the Bot.. So how can i de index my website completely from Google Index...?

[FranticFish]

Welcome to the forums!

1) fix your robots.txt (and remember to correct when you go live so that the site IS crawlable, I've seen this happen quite a few times)

2) you can use the 'remove' function in Webmaster Tools to remove a page or a whole folder, but if you want the exact same urls to be indexed once the site is ready to go live I'm not so sure that I'd remove them. If they're on separate urls from your 'live' site (i.e. /new/ or /dev/) then you can just clear out the whole folder.

3) next time, password-protect your test site :)

[phranque]

welcome to WebmasterWorld, patel_rosh!

password-protect your test site

if you add HTTP Basic Authentication to your development site's configuration, googlebot will get a 401 Unauthorized response and will deindex each url subsequently requested.

is your development site on a separate hostname?

[lucy24]

If the whole site has already been indexed, roboting it out may not do much good. (In this situation, would the existing snippet be replaced by "A description is not available..." or would they continue using what they've got?)

Option B is to let them continue crawling-- since they already know about you-- and slap on a noindex header. Doing it globally is probably simpler at this point than adding it to all existing content (including non-html material). The Apache form is

Header set X-Robots-Tag "noindex"

and I'm sure there is an equally simple command for IIS.

[petehall]

It is so easy. Just lock it down on IP and it will vanish.

Or does the whole world need to see it?