Forum Moderators: goodroi

Message Too Old, No Replies

Re-instate pages previously blocked

sorry for this noob question

         

dibbern2

9:45 pm on Jul 5, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I can't believe I've never had to do this before, but I need to re-instate pages that are presently blocked by my robots.txt file.

I realize that I can just remove the appropriate lines in the file, but I'm wondering what else I might do to boost crawlers' attention and get back in the serps...

There are dozens, so I don't think I should submit each page by hand. (Although I suppose I could.) Would a Google sitemap get things moving? Any other ideas?

goodroi

11:51 am on Jul 6, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Googlebot will request your robots.txt file each time it visits your site. If you make a change and remove the lines in robots.txt that are blocking those pages google will start noticing it the next time it visits your site.

In my experience Google will react very quickly with no additional work needed (but that is specific to each site). To increase google's indexing of your site you should get more links.

Draganfly

7:07 pm on Jul 16, 2007 (gmt 0)

10+ Year Member



I'm having the same issue:

I have a URL that used to be in my robots.txt file as "Disallow:" but now I want it indexed because I actually had that URL by mistake. I've checked my Google Webmaster Tools robots.txt analysis and Google has downloaded the latest version of our robots.txt file. It's been 2 weeks since I removed that URL from the robots file and it still isn't indexed by Google. The link is cross-linked on about 100 pages and in our blog. I don't understand why Google is still ignoring that URL.

goodroi

11:23 pm on Jul 16, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



hi draganfly,

it is also helpful to have other sites link to your content and not just internal links. you said that in google's webmaster tool it showed it had retrieved the updated version of robots.txt. did you also test robots.txt to make sure the page is truly unblocked?