Welcome to WebmasterWorld Guest from 3.80.6.254

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Will google still rank a site if blocked in Robots.txt

     
7:36 pm on Feb 17, 2014 (gmt 0)

Preferred Member

10+ Year Member

joined:July 10, 2005
posts:495
votes: 0


This is a question for a client.

He's a reseller for GoDaddy..and has his own white label with GoDaddy Reseller . But GoDaddy Reseller recommends everybody to keep their sites as is.

Which means this happens to their white label in Robots.txt

Disallow: /


Their main homepage is listed in the google index. (i'm guessing crawled before the robots.txt was implemented)

So is it possible to still get his home page ranked decently high in the search if enough positive white-hat marketing tactics were used ? and has decent legit backlinks back to his site ?
4:22 am on Feb 18, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15928
votes: 884


Is he trying to rank for the phrase "a description of this site is not available"?
8:57 am on Feb 18, 2014 (gmt 0)

New User

5+ Year Member

joined:Feb 18, 2014
posts: 3
votes: 0


Yes and yes.

I've done that with several sites, and it will still list the site - but just the root page (just the domain).
The snippet will say something like "a description of this site is not available b/c this site does not allow us to crawl it via its robots.txt"

But yes, the homepage will remain listed. Which sucks if you're trying to go low-key.
4:29 am on Feb 19, 2014 (gmt 0)

Preferred Member

10+ Year Member

joined:July 10, 2005
posts:495
votes: 0


interesting thx, weis.

and lol lucy. (soo sarcastic)
12:20 pm on Feb 19, 2014 (gmt 0)

New User

5+ Year Member

joined:Feb 19, 2014
posts:1
votes: 0


Their main homepage is listed in the google index. (i'm guessing crawled before the robots.txt was implemented)
<snip>

[edited by: goodroi at 1:27 pm (utc) on Feb 19, 2014]
[edit reason] Welcome to WebmasterWorld, now please go read & follow the community guidelines [/edit]

1:25 pm on Feb 19, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 28, 2003
posts:1979
votes: 0


Google will rank a site, even number one, if they are disallowing all in robots.txt. I recently saw an example if this and wrote about it on my blog.

It has been a few months and the site it still there, ranking number one. I am pretty sure it has to do with links, and not the content on the site.
9:58 pm on Feb 19, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15928
votes: 884


My test site is 100% roboted-out, but it gets periodic human visitors from Bing search. They seem to value exact name matches more highly than google does. (I got curious and searched for the same phrase in google. I was nowhere. Well, at least not in the top 30.)

But you'd have to have some absolutely phenomenal links to outrank sites where your search phrase actually occurs on the page.
2:13 am on Feb 20, 2014 (gmt 0)

Preferred Member

10+ Year Member

joined:July 10, 2005
posts:495
votes: 0



They seem to value exact name matches more highly than google does. (I got curious and searched for the same phrase in google. I was nowhere. Well, at least not in the top 30.)



This is a new thing with Google lately, combined with all their recent trashy algorithm changes. It's really getting bad and they must be completely blind if they aren't recognizing the thousands of complaints from webmasters lately.

Matt Cutts must be hiding in a corner somewhere again.

It doesn't matter at all anymore to them..if you have an exact phrase...and these exact phrase keywords are ALL that are in your domain name and title tag, and <h1, h2 tags....and other parts of your site for that matter. Bing will correctly rank you within the top 5 most of the time. But Google's sending everybody's site out to the sea. Except if your from a huge company, news, or whatever. Amazon, Ebay, News sites.

Google used to be the 'peoples' search engine. Now it's the corporates search engine. They must of come to terms with an under-the-table agreement somewhere to help the economy by boosting key companies rankings.
5:20 am on Feb 20, 2014 (gmt 0)

New User

5+ Year Member

joined:Feb 20, 2014
posts: 9
votes: 0


and if you want that in search you will see some content in description please remove this tag from your page. "NOOPD, NOYDIR" and your site will show your content or will get description from dmoz or ydir.