How long does it take Google to crawl back a new site

index a site

[findcosmet]

Hi all,

I am an inexperienced webmaster. We have this site set up for a bit over a month ago. I had checked Google with keyword mysite.com, and I found that Google indexed our site two months ago and have a under construction page cached. Does anyone have a rough idea how long it will take Google to come back? BTW, we have submitted our url to Google about two weeks ago. Google spider still has not come around. Is this normal?

Also, I would like to know if it is ok that we do not have robot.txt in our site, for I am afraid that I may make things worse. Does anyone know Google's crawler IP, so that I can track it in my server log?

Thanks,

[metrostang]

I put up a new domain on June 6 and 22 pages were crawled within 3 days. I had not submitted to Google. After another week I did submit the site and now all 63 pages have been crawled but are not included in any searches yet. I do have a robots file, but it's only to prevent all of the 404 errors.

[BillyS]

Welcome to webmasterworld!

As many here will tell you, the best way to get Googlebot to spider your site is to have an incoming link from a site already in their index.

There are lots of directories out there that will be happy to link to a quality site.

My experience with Google is that the visits initially are far apart, but if you change content frequently on your home page, the bot tends to revisit quickly. I use a stats package (AWStats) to follow Google's progress. It does so not by IP address (which are numerous), but by Agent name (Googlebot).

Finally, since you're new I would suggest the following as "must read."

[searchengineworld.com...]

Again, welcome.

[treeline]

If you want to be indexed by search engines you do not need robots.txt - it's purpose is to restrict what they can look at. If robots.txt isn't there, they assume they can look at whatever they want, which is usually good.

[Reid]

a month is not very long - googlebot may be coming back. it has under construction page cached. I would check the site with a robot sim to make sure it is not having a problem being crawled.
There are several googlebots with dozens of possible IP's - googlebot has no referrer string but will always id itself as 'googlebot'.

[Rollo]

If you want to get in right away, try to find a high PR link. You'll be indexed almost immediately.

I just launched a site that started with 3000 pages about a week and a half ago, so far Google has indexed about 250 pages, Yahoo 2 and MSN 1. Google is by far the fastest indexer around.

[Panacea]

sometimes 3 days, sometimes 3 weeks, sometimes 3 months

[findcosmet]

Thanks for all these helpful tips. I have a dynamic site (a Wiki). Even though, I have the "?" in the url replaced by "/", I was worry that spiders do not like it.

I finally saw Google came around. However, it did not crawl a lot of my pages. My site uses two levels of nested table, so that it has a master index menu on the left side and page content on the right. But Googlebot skips the links on the left and only picks up a couple of links in the content.

I am planning to have a master index page, which does not have the left side menu. In its content, it has links to the content pages. Then in my home page, I will add a visible link to this page as site map. Anyone knows if my design is ok?

Thanks.

[JoeHouse]

Does Googlebot like to see frequent changes to the homepage?

Does it help or hurt to make changes often on homepage?

Its my opinion its much better when googlebot visits homepage often verus not often. Am I correct to assume this?

Also does it help with rankings and traffic to change homepage often?

Thanks in advance for your response.

[Reid]

joehouse - I don't think google favors the homepage as far as ranking is concerned, it just gets rated highest because of the linking structure, as the root of the site.
One of your other pages could outrank your homepage.
How often to change content - depends on what you change it to. Hopefully to add links to the menu.
When you update a page the spider will re-crawl it for sure but the ranking could go up or down depending on the content.
I found a hompage that I like and then went working on other areas of the site for a while, when I get tired of looking at it or need to, I will change it again but to keep changing it just for the sake of ranking will not produce the desired results.

[uk_webber]

Get a link from a PR5 + page and it will be days. Works for me every time.

[findcosmet]

I am not sure if it is because my site index strategy is working or if it is because my site has been there longer. Today, I saw in the server log,

"Yahoo! Slurp; [help.yahoo.com...] I assume this is Yahoo's crawler.

It seems that it has crawed a lot more pages than before.