homepage Welcome to WebmasterWorld Guest from 54.204.64.152
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
how long for robots.txt removal?
johnnyb3




msg:184408
 6:59 pm on Dec 4, 2003 (gmt 0)

Yesterday I put a robots.txt that basically disallowed the SE's from spidering or indexing my site through:

User-agent: *
Disallow: /

How long will it take for a site to be removed from google this way? I've been considering the emergency removal, and I'll probably end up doing it if it takes more than a few days.

 

nakulgoyal




msg:184409
 3:43 am on Dec 5, 2003 (gmt 0)

Well, what I feel is this won't remove the currently indexed pages so easily. An idea that popped out of me is download your website on your machine, change the filenames and do a simple find and replace and upload them to your website. Make sure you delete the old files.

This way your website pages won't be find. What do you say?

TheDave




msg:184410
 3:48 am on Dec 5, 2003 (gmt 0)

Even though google won't crawl a page that is disallowed with robots.txt, they can still list a link to your site. If you want to ensure complete removal of your site, I would recommend not using robots.txt, but rather using meta tags on each page. Let the bots crawl the content, but don't let them index it. This keeps google from listing links to pages you don't want it listing links to.

Krapulator




msg:184411
 4:42 am on Dec 5, 2003 (gmt 0)

To answer your question though: probably up to a month - in the worst case a few months.

GoogleGuy




msg:184412
 4:57 am on Dec 5, 2003 (gmt 0)

I think it should be much sooner than a month. If it were an office pool, I'd bet on less than 10-14 days. Let us know when you start to see pages leaving (it will probably be a gradual process as we try to fetch each page and find the robots.txt).

TheDave




msg:184413
 4:58 am on Dec 5, 2003 (gmt 0)

But GoogleGuy, won't you still be listing links to his site, or does that only work if the link is found on the site that it's refering to?

Krapulator




msg:184414
 4:59 am on Dec 5, 2003 (gmt 0)

I stand corrected :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved