| 4:09 pm on Feb 20, 2014 (gmt 0)|
welcome to WebmasterWorld, andrewc!
|Here is the message form the Blocked URLs section: |
that would be normal if you have Disallow directives in your robots.txt file
| 4:28 pm on Feb 20, 2014 (gmt 0)|
I used to have them, now i removed them. Could it be related to the URLs i removed in the last two days from the google index? (around 200)
| 11:04 pm on Feb 20, 2014 (gmt 0)|
Well, it's possible they sent you the wrong form letter :) Crawling and indexing are separate activities. It's also possible that if you remove a lot of URLs at once, they take an extra look at robots.txt to see if anything there has changed.
What did happen to those 200 pages? Are they roboted-out, physically removed (404 or 410), or does each one have a <noindex> tag?
| 11:18 pm on Feb 20, 2014 (gmt 0)|
I'm just looking at two terms: robots.txt and Sitemap... Not the same things... What's up?
| 12:29 am on Feb 21, 2014 (gmt 0)|
In Google terminology blocking means exclude from crawling.
removing a URL from the index doesn't block it.
| 6:49 am on Feb 21, 2014 (gmt 0)|
@lucy24 we did a redesign and some of the urls are old categories that don't exist anymore. Since last week are physically removed (410) but i also removed them from google urls to speed things.
| 8:40 am on Feb 21, 2014 (gmt 0)|
Good. If it doesn't exist, they can't possibly change their minds three months down the line. (Or can they? Has anyone ever found a search engine bringing back pages that it hasn't crawled in months?)
|I'm just looking at two terms: robots.txt and Sitemap |
I think it means that the robots.txt file includes references to six (!) sitemaps. It isn't part of the robots.txt standard-- well, nothing is except "Disallow" --but it's to google's advantage to recognize it ;)
Speaking of which: Make double-sure that no sitemap, anywhere, mentions those old pages.
| 4:48 pm on Feb 24, 2014 (gmt 0)|
Ok, cleared the robots.txt
Now it looks like this: User-agent: *
The problem is that there are some blocked urls, at least google says this. How long should it take to clear those blocked urls from WMT?
| 9:34 pm on Feb 24, 2014 (gmt 0)|
It should happen right away if you "fetch as googlebot". But don't say
(nothing after "Disallow:")
If you can do something within the strictest confines of the robots.txt standard, do so.
| 8:09 pm on Feb 25, 2014 (gmt 0)|
Ok, i just updated the robots. txt. Thanks for the tip lucy24!