Welcome to WebmasterWorld Guest from 54.147.217.76

Forum Moderators: open

Message Too Old, No Replies

how long for robots.txt removal?

   
6:59 pm on Dec 4, 2003 (gmt 0)

10+ Year Member



Yesterday I put a robots.txt that basically disallowed the SE's from spidering or indexing my site through:

User-agent: *
Disallow: /

How long will it take for a site to be removed from google this way? I've been considering the emergency removal, and I'll probably end up doing it if it takes more than a few days.

3:43 am on Dec 5, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Well, what I feel is this won't remove the currently indexed pages so easily. An idea that popped out of me is download your website on your machine, change the filenames and do a simple find and replace and upload them to your website. Make sure you delete the old files.

This way your website pages won't be find. What do you say?

3:48 am on Dec 5, 2003 (gmt 0)

10+ Year Member



Even though google won't crawl a page that is disallowed with robots.txt, they can still list a link to your site. If you want to ensure complete removal of your site, I would recommend not using robots.txt, but rather using meta tags on each page. Let the bots crawl the content, but don't let them index it. This keeps google from listing links to pages you don't want it listing links to.
4:42 am on Dec 5, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



To answer your question though: probably up to a month - in the worst case a few months.
4:57 am on Dec 5, 2003 (gmt 0)

WebmasterWorld Senior Member googleguy is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I think it should be much sooner than a month. If it were an office pool, I'd bet on less than 10-14 days. Let us know when you start to see pages leaving (it will probably be a gradual process as we try to fetch each page and find the robots.txt).
4:58 am on Dec 5, 2003 (gmt 0)

10+ Year Member



But GoogleGuy, won't you still be listing links to his site, or does that only work if the link is found on the site that it's refering to?
4:59 am on Dec 5, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I stand corrected :)