Welcome to WebmasterWorld Guest from 3.226.251.205

Forum Moderators: goodroi

Message Too Old, No Replies

Allowing online seo analytics tools to crawl my website?

     
11:15 pm on Oct 24, 2015 (gmt 0)

New User

joined:Oct 20, 2015
posts:8
votes: 0


I have disallowed all the bots to crawl my website because it is under construction.
What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.
So what changes should I make to my robots.txt, so these seo analytics tools can crawl it.
The website has some important company data they don't want anyone to access, that's why i can't allow all.

Error
This URL cannot be reviewed. There may be several reasons for this: the URL does not exist, the website does not allow WooRank to make reviews, the website denies both our HEAD and GET requests, the DNS sends to an infinite loop or the robots.txt page does not allow access to our bot.
11:59 pm on Oct 26, 2015 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9074
votes: 6


The website has some important company data they don't want anyone to access, that's why i can't allow all.


If you are publishing the data on the web without at least a password to protect it, it will be accessed and archived, whatever you put in your robots.txt file. A robots.txt is only respected by those who want to respect it.

So either password-protect your site or take it offline until it's ready.

As for the error message, it is self-explanatory.
12:50 am on Oct 27, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15753
votes: 826


I have disallowed all the bots to crawl my website because it is under construction.
What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.

I don't understand how you get from "under construction" to "want to allow seo analytics tools". If it's under construction, nobody should be crawling, except maybe a link checker.