Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Removing robots.txt Exclusion

How long to get back in Google?



10:39 am on Nov 16, 2004 (gmt 0)

10+ Year Member

need help about my site back in google.
we have 4 different domain for the same site and all have same content and everything so it was making duplication of our website so i suggested to add either robots.txt or to redirect every domain to main site and finally it was decied to add robots.txt and our system admin did big mistake of adding robotx.txt file to main site which was made to stop all the spiders crawling other domains, and it was there for week .
so google removed our site from database.. now even when i do URL search nothing comes up, i lost PR which i achived so far, and everything is gone.

i need your help in this , if you have any tips related to how to get site back in google soon..
please give me...


11:22 am on Nov 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I'm no robots.txt expert, but your situation sounds like it could be more than simply a robots.txt problem.

Hopefully others will chime in...


12:15 pm on Nov 16, 2004 (gmt 0)

WebmasterWorld Administrator 10+ Year Member

Just remove the robots.txt from the original domain and wait a while until Gbot comes back. You should return to the index pretty fast imo.


7:45 am on Nov 17, 2004 (gmt 0)

10+ Year Member


i removed it. but my site is completely removed from google..so i just want to any ways to come back faster , and want to know any idea when will i get my page rank back. which was 4


9:02 am on Nov 19, 2004 (gmt 0)

10+ Year Member

I made the same mistake back in July. Everything gone! I understand that google will return in 90 days to recheck the robots.txt file. My 90 days have come and gone but still no luck.

Patrick Taylor

11:39 am on Nov 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

if you have any tips related to how to get site back in google soon

If you had four versions of identical pages, each on its own domain, then Google could have seen this as a breach of its guidelines in each of the four cases instead of only three. In other words, the four domains are tainted. I once inadvertently left a test version of a site on a different domain to the main one. At one point the main one enjoyed PR4, but then went down to PR1. This was almost a year ago, and months after removing the test site the main one remains at PR1 with virtually no traffic from Google. In fact it's only kept online out of curiosity over how long, if ever, it regains its ranking.

In your case I would be inclined to start again with a fresh domain, acquire whatever external links to it you can, and just wait. Maybe even alter the content so that there is less chance of a connection made between it and those four earlier ones.

Hard to swallow, I would imagine, but my advice would be just to get on with it instead of waiting and hoping.


12:32 pm on Nov 19, 2004 (gmt 0)

10+ Year Member

hi there

A google ban could be in place. Duplicate content is a violation of googles TOS.

Have you run a ranking report on the other engines?

You should try to resubmit to Google.



1:02 pm on Nov 19, 2004 (gmt 0)

10+ Year Member

i did all possible thing i rank well in yahoo. i'm there in all other search engines google visited my other domains in month of september and october not after that
and my main domaim was last crawl by google was on 2nd November and same day that robots.txt was added, actually, the problem happen cos all my four domains are pointing to one same ip.so what ever changes i do to one will reflect all four...


1:32 pm on Dec 4, 2004 (gmt 0)

10+ Year Member

Can you send me your robots.txt file and who wrote it?

It sounds like it blocked googlebot from crawling a folder or set of files.



5:03 pm on Dec 4, 2004 (gmt 0)

methinks you got thrown out for duplicate content. I have robots.txt on many domains, that block Google completely and even block it via rewrite.

The files already indexed (months ago) are still in and when I remove the robots and rewrite within a wek or two it gets indexed again.

maybe the robots.txt placement and this removal were just coincidental. Of course, unless they're google no one knows for sure.


5:54 pm on Dec 4, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

googles TOS

Accepting duplicate content, auto generated content, and long lists of keywords are all part of my Site's TOS that every search engine implicitly accepts when they crawl my site and take my content for free.

I think it is about time Google started to respect it.


6:49 am on Dec 7, 2004 (gmt 0)

10+ Year Member

if you were in the index previously and you didn't change anything except add the robots.txt for a week. Removal will get you back in the index sooner or later.

I suggest you use the add url form at google. Even it doesn't benefit you. For sure it won't hurt you.

Another thing. Visit your page with opera browser with adsense turned on. mediabot most likely will step by to check your robots.txt


7:41 am on Dec 7, 2004 (gmt 0)

10+ Year Member

ok but i dont have opera browser and due to security resons i'm not allowed to donload anything... is ther any other way except opera browser.

as i'm there in google adwords, and all other serach engiens except google..



3:51 pm on Dec 7, 2004 (gmt 0)

10+ Year Member

well opera runs adwords on their browser. The same as people running adwords on their homepage to make a couple extra $$$ pocketmoney.

Google can only serve relevant ads if they know the content. Since you are not in the index the only thing that can happen is that the mediabot will visit to get the information.


Featured Threads

Hot Threads This Week

Hot Threads This Month