Forum Moderators: phranque
If I do this:
User-agent: *
Disallow: /
Later on, when I want bots to find my site, if I remove the '/' will bots 'return', or will I have to take more proactive steps to make sure they 'find' my site.
Thanks, sorry if this is covered somewhere else, I did look.
The best way I've found is to place the site I'm working on in another directory where the bots can't go. Then, keep an under construction page in the root along with a skleton of the other pages. When the bots find them, they will keep coming back, even if they are under const pages. Then, when I'm ready, I move everything over to go live and viola! The new content is indexed very quickly because the bots are already looking at the right pages.
You can also do what you are doing. Using the robots.txt file will prevent the spiders from entering and as long as you have links pointing to your website, the search engines will regularly visit your site trying to index your content.
there might be some issue with duplicate content
Just make sure that each page has a little description of what's to come with some targeted keyphrases and says at the end that you are working to provide the best information possible and that this page will up up soon with relavant info. Make sure each page is a little different and you should be good!
