homepage Welcome to WebmasterWorld Guest from 54.237.134.62
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
How long does it take Google to crawl back a new site
index a site
findcosmet




msg:705127
 2:16 am on Jun 22, 2005 (gmt 0)

Hi all,

I am an inexperienced webmaster. We have this site set up for a bit over a month ago. I had checked Google with keyword mysite.com, and I found that Google indexed our site two months ago and have a under construction page cached. Does anyone have a rough idea how long it will take Google to come back? BTW, we have submitted our url to Google about two weeks ago. Google spider still has not come around. Is this normal?

Also, I would like to know if it is ok that we do not have robot.txt in our site, for I am afraid that I may make things worse. Does anyone know Google's crawler IP, so that I can track it in my server log?

Thanks,


 

metrostang




msg:705128
 12:54 am on Jun 23, 2005 (gmt 0)

I put up a new domain on June 6 and 22 pages were crawled within 3 days. I had not submitted to Google. After another week I did submit the site and now all 63 pages have been crawled but are not included in any searches yet. I do have a robots file, but it's only to prevent all of the 404 errors.

BillyS




msg:705129
 12:58 am on Jun 23, 2005 (gmt 0)

Welcome to webmasterworld!

As many here will tell you, the best way to get Googlebot to spider your site is to have an incoming link from a site already in their index.

There are lots of directories out there that will be happy to link to a quality site.

My experience with Google is that the visits initially are far apart, but if you change content frequently on your home page, the bot tends to revisit quickly. I use a stats package (AWStats) to follow Google's progress. It does so not by IP address (which are numerous), but by Agent name (Googlebot).

Finally, since you're new I would suggest the following as "must read."

[searchengineworld.com...]

Again, welcome.

treeline




msg:705130
 12:59 am on Jun 23, 2005 (gmt 0)

If you want to be indexed by search engines you do not need robots.txt - it's purpose is to restrict what they can look at. If robots.txt isn't there, they assume they can look at whatever they want, which is usually good.

Reid




msg:705131
 8:21 pm on Jun 23, 2005 (gmt 0)

a month is not very long - googlebot may be coming back. it has under construction page cached. I would check the site with a robot sim to make sure it is not having a problem being crawled.
There are several googlebots with dozens of possible IP's - googlebot has no referrer string but will always id itself as 'googlebot'.

Rollo




msg:705132
 1:28 pm on Jun 24, 2005 (gmt 0)

If you want to get in right away, try to find a high PR link. You'll be indexed almost immediately.

I just launched a site that started with 3000 pages about a week and a half ago, so far Google has indexed about 250 pages, Yahoo 2 and MSN 1. Google is by far the fastest indexer around.

Panacea




msg:705133
 2:27 pm on Jun 24, 2005 (gmt 0)

sometimes 3 days, sometimes 3 weeks, sometimes 3 months

findcosmet




msg:705134
 6:02 pm on Jun 24, 2005 (gmt 0)

Thanks for all these helpful tips. I have a dynamic site (a Wiki). Even though, I have the "?" in the url replaced by "/", I was worry that spiders do not like it.

I finally saw Google came around. However, it did not crawl a lot of my pages. My site uses two levels of nested table, so that it has a master index menu on the left side and page content on the right. But Googlebot skips the links on the left and only picks up a couple of links in the content.

I am planning to have a master index page, which does not have the left side menu. In its content, it has links to the content pages. Then in my home page, I will add a visible link to this page as site map. Anyone knows if my design is ok?

Thanks.

JoeHouse




msg:705135
 6:39 pm on Jun 24, 2005 (gmt 0)

Does Googlebot like to see frequent changes to the homepage?

Does it help or hurt to make changes often on homepage?

Its my opinion its much better when googlebot visits homepage often verus not often. Am I correct to assume this?

Also does it help with rankings and traffic to change homepage often?

Thanks in advance for your response.

Reid




msg:705136
 3:40 am on Jun 26, 2005 (gmt 0)

joehouse - I don't think google favors the homepage as far as ranking is concerned, it just gets rated highest because of the linking structure, as the root of the site.
One of your other pages could outrank your homepage.
How often to change content - depends on what you change it to. Hopefully to add links to the menu.
When you update a page the spider will re-crawl it for sure but the ranking could go up or down depending on the content.
I found a hompage that I like and then went working on other areas of the site for a while, when I get tired of looking at it or need to, I will change it again but to keep changing it just for the sake of ranking will not produce the desired results.

uk_webber




msg:705137
 2:43 pm on Jun 26, 2005 (gmt 0)

Get a link from a PR5 + page and it will be days. Works for me every time.

findcosmet




msg:705138
 10:04 pm on Jun 27, 2005 (gmt 0)

I am not sure if it is because my site index strategy is working or if it is because my site has been there longer. Today, I saw in the server log,

"Yahoo! Slurp; [help.yahoo.com...] I assume this is Yahoo's crawler.

It seems that it has crawed a lot more pages than before.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved