Updating robots.txt and Crawlers behaviour

How to tell bots to come back when you changed robots.txt banning them

[silverbytes]

I discovered that robots.txt file was using were banning all bots! I changed that using a known working and allowing robots.txt file. I already uploaded the new robots.txt file ... what can I do now? How do I invite all bots again to crawl my site? Google bot will come again and just recognize that can crawl or what?

[Sanenet]

Most of the little blighters should just come back on their own terms and recheck. Play the waiting game...

Also, a few new inbound links shouldn't harm either.

[silverbytes]

Does that include googlebot?

[sordinous]

Yes, that includes Googlebot as well. Every time he starts to crawl your site he is asking for your robots.txt-file.
But, what i've seen is, folders first time visited by Googlebot and later on by robots.txt disallowed are remaining in the google-directory for a very long time. Especially if the data still remains in the folder. Has anyone here a solution or idea for that?

With best regards,
Sordinous

[Sanenet]

G. has had so many problems with people asking to remove pages from their SERPS that they now remove robots.txt banned pages from their DB for a minimum of 6 months (Apparantly, see threads currently being run on this forum for more info).

Try moving those banned pages to a new directory theme and adding new inbound links directly through to them.

[Reid]

There are 2 things I would try

submit your 'good' (and validated) robots.txt to the URL removal console. If you submitted a robots.txt banning entire site then yes, you would have to wait 6 months but if googlebot found the bad robots.txt on it's own then submitting a good robots.txt might clear the robots.txt google has on record. One person on WW even claimed that he gets googlebot coming right away (no inbound links, he's tried it) by submitting an empty robots.txt to the removal console.

if that doesn't work within 5 days then I would submit a 'google sitemap' and see if that works. if not theres always a 'reinclusion request' if a canned response will help you sleep.