homepage Welcome to WebmasterWorld Guest from 54.211.100.183
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
how long for robots.txt removal?
johnnyb3

10+ Year Member



 
Msg#: 20334 posted 6:59 pm on Dec 4, 2003 (gmt 0)

Yesterday I put a robots.txt that basically disallowed the SE's from spidering or indexing my site through:

User-agent: *
Disallow: /

How long will it take for a site to be removed from google this way? I've been considering the emergency removal, and I'll probably end up doing it if it takes more than a few days.

 

nakulgoyal

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 20334 posted 3:43 am on Dec 5, 2003 (gmt 0)

Well, what I feel is this won't remove the currently indexed pages so easily. An idea that popped out of me is download your website on your machine, change the filenames and do a simple find and replace and upload them to your website. Make sure you delete the old files.

This way your website pages won't be find. What do you say?

TheDave

10+ Year Member



 
Msg#: 20334 posted 3:48 am on Dec 5, 2003 (gmt 0)

Even though google won't crawl a page that is disallowed with robots.txt, they can still list a link to your site. If you want to ensure complete removal of your site, I would recommend not using robots.txt, but rather using meta tags on each page. Let the bots crawl the content, but don't let them index it. This keeps google from listing links to pages you don't want it listing links to.

Krapulator

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 20334 posted 4:42 am on Dec 5, 2003 (gmt 0)

To answer your question though: probably up to a month - in the worst case a few months.

GoogleGuy

WebmasterWorld Senior Member googleguy us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 20334 posted 4:57 am on Dec 5, 2003 (gmt 0)

I think it should be much sooner than a month. If it were an office pool, I'd bet on less than 10-14 days. Let us know when you start to see pages leaving (it will probably be a gradual process as we try to fetch each page and find the robots.txt).

TheDave

10+ Year Member



 
Msg#: 20334 posted 4:58 am on Dec 5, 2003 (gmt 0)

But GoogleGuy, won't you still be listing links to his site, or does that only work if the link is found on the site that it's refering to?

Krapulator

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 20334 posted 4:59 am on Dec 5, 2003 (gmt 0)

I stand corrected :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved