Forum Moderators: goodroi

Message Too Old, No Replies

Allowing online seo analytics tools to crawl my website?

         

janiman

11:15 pm on Oct 24, 2015 (gmt 0)

10+ Year Member



I have disallowed all the bots to crawl my website because it is under construction.
What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.
So what changes should I make to my robots.txt, so these seo analytics tools can crawl it.
The website has some important company data they don't want anyone to access, that's why i can't allow all.

Error
This URL cannot be reviewed. There may be several reasons for this: the URL does not exist, the website does not allow WooRank to make reviews, the website denies both our HEAD and GET requests, the DNS sends to an infinite loop or the robots.txt page does not allow access to our bot.

encyclo

11:59 pm on Oct 26, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The website has some important company data they don't want anyone to access, that's why i can't allow all.


If you are publishing the data on the web without at least a password to protect it, it will be accessed and archived, whatever you put in your robots.txt file. A robots.txt is only respected by those who want to respect it.

So either password-protect your site or take it offline until it's ready.

As for the error message, it is self-explanatory.

lucy24

12:50 am on Oct 27, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I have disallowed all the bots to crawl my website because it is under construction.
What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.

I don't understand how you get from "under construction" to "want to allow seo analytics tools". If it's under construction, nobody should be crawling, except maybe a link checker.