Do search engines, Google in particular, need access to my wp-admin area? They demoted me from 5 stars for SEO to 2 stars because they cannot access that area. If they need access, how do I allow it? I use Yoast. Thanks for your help.
travelin cat
1:13 am on May 6, 2014 (gmt 0)
The search engines do not need access to your wp-admin directory. In fact, it's good practice to password protect them.
Which stars are you referring to? And how do you correlate the two?
azlinda
6:41 am on May 6, 2014 (gmt 0)
The "stars" are the five blue dots in the Google AdSense dashboard. I have had five blue dots until I found the following message: Crawler errors - Robot denied for [..........] and [.......]
azlinda
6:47 am on May 6, 2014 (gmt 0)
Well, that didn't work out well, did it.
robot denied for:
wp-admin wp-admin/customize.php
lorax
11:32 am on May 6, 2014 (gmt 0)
Is Google Bot denied via the htaccess?
azlinda
5:56 pm on May 6, 2014 (gmt 0)
No, Google Bot is indexing the site. It seems their problem is having no access to the admin area.
not2easy
6:37 pm on May 6, 2014 (gmt 0)
You have the /wp-admin/ folder Disallowed in robots.txt, right? That is all you really need to do - and ignore their pleas, there is nothing in there they need. They will get over it.
lucy24
9:08 pm on May 6, 2014 (gmt 0)
No, Google Bot is indexing the site.
I thought he meant, is the googlebot blocked from that specific directory. But it shouldn't make any difference, because if the robots.txt block is correctly worded-- which it obviously is-- then search engines will never ask for the page in the first place.
What worries me more is why they're asking for pages in this directory. Surely you don't have visible links to private content?
lorax
12:36 pm on May 7, 2014 (gmt 0)
@not2easy @lucy24 - that's what I'm wondering. Robots should get rid of the issue but I've never seen Googlebot even request it on my sites. Maybe the sitemap file includes it?
not2easy
4:04 pm on May 7, 2014 (gmt 0)
Oh, I just remembered something - after the issue of the responsive version of WP, when auto-updates started, I saw a similar weird note in the AdSense UI - Seems they wanted to access wp-admin/css/colors/*.scss. Because this "error" is not due to Googlebot or any search engines, but an evaluation process within AdSense, it really isn't controlled in robots.txt or GWT.
I'm not saying that is the issue for the OP, but given that this is about Stars in the AdSense interface it could be. They seem to place a good deal of importance on responsive issues. To see what specific files they want to look at in wp-admin, all you need to do is click on the warning. The problem is not related to search engines or crawling, only to pleasing the responsiveness checking component of AdSense.
