Robots.txt is set up correctly but GSC not crawling?

[wickles]

Some of my main pages that have been published on the site for several years with no issues are now showing up as "Indexed, though blocked by robots.txt". Nothing has changing. Not sure why this is happening. I looked at the robots tester and all the pages being flagged are validated. Also all the urls are on the sitemap. Nothing is marked as noindex, nofollow. Not sure what else to check?

[not2easy]

We have seen lots of people with this problem over the past weeks. You don't need to do anything, they will take care of it, it is their problem.

[phranque]

when you see those pages in the index, do they look like they have been blocked by robots.txt?

[lucy24]

Or, for that matter: according to site logs, have the URLs in fact been crawled?

[wickles]

Searches on Google are showing no information but the URL. I also requested for the pages to validate the fix and now more pages are showing that there is an error. Traffic is now way down and keyword rank is lost.

[edited by: wickles at 1:43 pm (utc) on Jul 20, 2022]

[wickles]

I can no longer "Test Live URL" on URL Inspection or "Request Indexing" on any of my pages.

[not2easy]

There is a limit to the number of rendering requests Google will allow, they do not encourage people to use that method to request indexing.

[wickles]

I know there is a limit of 10 request, but I can't even get one to go through.

[lucy24]

but I can't even get one to go through

What, exactly, do you mean by “can’t get it to go through”? (This is a question more often asked in the technical subforums such as Apache.)

To repeat an earlier question: What googlebot activity do the site's access logs show?

[tangor]

If it says "blocked by robots.txt" there is something in your robots.txt that is blocking SOME part of g --- most likely regarding images or perhaps some ua used by g that does not SAY g.

Check your site logs.