homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Webmaster General
Forum Library, Charter, Moderators: phranque

Webmaster General Forum

Google Reset?
How to tell google to start spidering my site from scratch?

 10:28 pm on Apr 6, 2008 (gmt 0)

I am almost finished building my site and google has already started spidering it, with lots of pages that I do not want google to know about (the site shares a database of products with my main site and this one is going to be a niche site only).

How do I tell google to reset my site and start spidering from lets say a week from now when I actually want the new site to go live?



 11:23 pm on Apr 6, 2008 (gmt 0)

Google aren't a 'webmaster service' that allows you to press a reset button and start all over whenever you make a mistake.

However you can use the available tools at hand:

1) Google Webmaster Tools [google.com] ('URL removal' tool) allows you to remove indexed URLs for up to 6 months (Google only).

Read the guideline in the Webmaster Tools to learn more of how to remove URLs by the use of this tool (for example a criteria for the tool to work is that the removed pages returns a 404 (Not Found) or a 410 (Gone).)

2) Use robots.txt [robotstxt.org] to disallow indexation of your website:

User-agent: *
Disallow: /

This prevent Search Engines (those who obeys the robots.txt though) to spider and index your website. It's not 100% safe but it should work.

3) Besides the robots.txt, you can also use the noindex meta-element: <meta name="robots" content="noindex">

Remember to remove the Disallow: / (you only need to remove the '/' to make your site available for the SEs) and the robots meta element when your pages are ready and you want the Search Engines to spider your website.

To prevent future unwanted indexation of URLs you should make a plan for how to launch new pages. For example you can take the mentioned steps as written above and/or you could make them password protected to prevent spiders and users from visiting your pages.


 11:32 pm on Apr 6, 2008 (gmt 0)

How do I tell google to reset my site and start spidering from lets say a week from now when I actually want the new site to go live?

If Google (and other SEs) allready have spidered and indexed your URLs, I don't think you can remove the wanted URLs from the index in a week.

Often it takes quite a while for Google and other SEs to remove URLs.


 11:52 pm on Apr 6, 2008 (gmt 0)

Thanks for the info,
I am 404ing the unwanted pages.


 4:24 am on Apr 7, 2008 (gmt 0)

You might also consider the little used <meta name="robots" content="noarchive" /> tag if you don't want old copies of your pages lingering around the internet.

The tag gets rid of the link "cached" under your google search results.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Webmaster General
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved