Welcome to WebmasterWorld Guest from 35.172.195.49

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

How to de index development version of my website?

     
11:37 am on Jan 8, 2015 (gmt 0)

New User

joined:Jan 8, 2015
posts: 3
votes: 0


Hey all..:) I am in a spot of bother a bit.. Actually my website is in development mode but i forget to put robots.txt file in it so what happened that my website under developed pages have been crawled by the Bot.. So how can i de index my website completely from Google Index...?
12:16 pm on Jan 8, 2015 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 16, 2009
posts:1087
votes: 83


Welcome to the forums!

1) fix your robots.txt (and remember to correct when you go live so that the site IS crawlable, I've seen this happen quite a few times)

2) you can use the 'remove' function in Webmaster Tools to remove a page or a whole folder, but if you want the exact same urls to be indexed once the site is ready to go live I'm not so sure that I'd remove them. If they're on separate urls from your 'live' site (i.e. /new/ or /dev/) then you can just clear out the whole folder.

3) next time, password-protect your test site :)
2:10 pm on Jan 8, 2015 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11874
votes: 245


welcome to WebmasterWorld, patel_rosh!


password-protect your test site

if you add HTTP Basic Authentication to your development site's configuration, googlebot will get a 401 Unauthorized response and will deindex each url subsequently requested.

is your development site on a separate hostname?
5:15 pm on Jan 8, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15937
votes: 889


If the whole site has already been indexed, roboting it out may not do much good. (In this situation, would the existing snippet be replaced by "A description is not available..." or would they continue using what they've got?)

Option B is to let them continue crawling-- since they already know about you-- and slap on a noindex header. Doing it globally is probably simpler at this point than adding it to all existing content (including non-html material). The Apache form is
Header set X-Robots-Tag "noindex"

and I'm sure there is an equally simple command for IIS.
6:31 pm on Jan 8, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 19, 2003
posts:865
votes: 3


It is so easy. Just lock it down on IP and it will vanish.

Or does the whole world need to see it?