Welcome to WebmasterWorld Guest from 184.108.40.206
I am an inexperienced webmaster. We have this site set up for a bit over a month ago. I had checked Google with keyword mysite.com, and I found that Google indexed our site two months ago and have a under construction page cached. Does anyone have a rough idea how long it will take Google to come back? BTW, we have submitted our url to Google about two weeks ago. Google spider still has not come around. Is this normal?
Also, I would like to know if it is ok that we do not have robot.txt in our site, for I am afraid that I may make things worse. Does anyone know Google's crawler IP, so that I can track it in my server log?
As many here will tell you, the best way to get Googlebot to spider your site is to have an incoming link from a site already in their index.
There are lots of directories out there that will be happy to link to a quality site.
My experience with Google is that the visits initially are far apart, but if you change content frequently on your home page, the bot tends to revisit quickly. I use a stats package (AWStats) to follow Google's progress. It does so not by IP address (which are numerous), but by Agent name (Googlebot).
Finally, since you're new I would suggest the following as "must read."
I finally saw Google came around. However, it did not crawl a lot of my pages. My site uses two levels of nested table, so that it has a master index menu on the left side and page content on the right. But Googlebot skips the links on the left and only picks up a couple of links in the content.
I am planning to have a master index page, which does not have the left side menu. In its content, it has links to the content pages. Then in my home page, I will add a visible link to this page as site map. Anyone knows if my design is ok?
Does it help or hurt to make changes often on homepage?
Its my opinion its much better when googlebot visits homepage often verus not often. Am I correct to assume this?
Also does it help with rankings and traffic to change homepage often?
Thanks in advance for your response.
"Yahoo! Slurp; [help.yahoo.com...] I assume this is Yahoo's crawler.
It seems that it has crawed a lot more pages than before.