Don't quite understand all the different googlebot's thingie

[Tiemen]

Oke, last month google crawled my website, but I forgot to put the right meta tags (index, follow) in the php section of the site, which happens to be almost the entire site. This way only about 5 pages were indexed.
In my stats I see that google visits my website 5 times exactly every day since than, while I did change the php section so it should as well index that... I think.
Now, let me get this right. I read some topics and if i'm right it is the freshbot checking up my pages (daily). So all I have to do is wait voor the deepcrawler to visit my site again and I will be indexed? This is right or not? Please explain to me :)

Grz - Tiemen

[Susanne]

Hello Tiemen, and welcome to WebmasterWorld :)
You don't need the index, follow in your pages. The default behaviour of spiders/robots is to index your pages and to follow their links.
Yes, you can expect to get into the Google index either within days (the next update is coming soon!) or end of April. Good luck!

[MetropolisRobot]

Good morning tieman.

There are two main googlebots (always afraid that there might be more)

freshbot (IP starts with 64) comes around often to sites that have fresh content. Its like a back scratching relationship. Google wants fresh content, you want spidering. If google sees your site as a source of fresh content then you'll see freshbot. AND sometimes during the month (roughly a month, before I get flamed for that comment) you may move about in the rankings if your content has changed a great deal etc etc. This move has never amounted to more than +1/-1 positions for me.

deepbot is the visitor you want (IP starts with 216). This chap comes along and reads your pages in preparation for the next google dance. This is the update that happens roughly once a month and causes webmasters great consternation when it happens late. This is where you see how your content, your page setup, your site setup, your inward links and PR rate against other sites for certain search keywords etc.

And finally, being spidered is no guarantee of appearing in the index. Also it's no guarantee that your pageranks will be good (so don't kick yourself too much over missing meta tags etc, this all takes time. I had a PR of zero for 2 months because of crimes against SE). As your site matures you'll get ample opportunity to learn from this site and others the good ways of structuring your site to achieve better SE ratings. You'll also learn what to avoid.

[Susanne]

"And finally, being spidered is no guarantee of appearing in the index." Right!
Oh yes, I forgot to tell you that your site must have links from other sites to make it into Google's index.

[Tiemen]

Well, there are enough links to the main page, and this was in indexed as well. What I'm looking for is google to index the rest of the site. If the meta tags aren't nessecary, then why isn't the rest of the site indexed? As I see it now google indexed only 2 levels...

[MetropolisRobot]

Tieman.

Patience.

I have a site that Google came for months and would only spider 50 or so pages then stopped.

The only way to have overnight success is to cheat. It's a sure way to get your site blacklisted faster than you can blink.

BTW I am not a patient person, but I am learning that patience is a virtue when dealing with all this stuff.

If I had one word of advice that would be this. Be ready to make changes after the Google Dance happens as deepbot comes soon after that. So if you see issues with your site etc after this dance, and want to have them incorporated into the next dance, you have to do them before deepbot gets to your site and trust me, it comes pretty quickly after the dance is over. You only have a limited window of opportunity to see the new results and react to them.

[Tiemen]

May be interesting that google didn't visit my site today yet, while most of the month it already visited my site in the early hours of the day... update coming?

[MetropolisRobot]

Heh, don't start that thread. It is not time. You'll know when the update is coming, and if it does not come on time, for sure.

Nope, sometimes Google comes along and sometimes not. If the IP is 64 (freshbot) then you may see if for days then not for days.

And the update comes when Google says it comes and not before.

[Susanne]

Tiemen,
It's always good to have a site map, both for humans and for spiders. Do you have one? If not, make a page that only contains spider friendly links to all the pages of your site. Make the links keyword rich. You can also, if you wish, add a sentence to each link that describes where the users will land if they follow that link. Finally, place a text link to the site map on your home page.

[freejung]

Tiemen, all of this sounds quite normal to me and nothing to be alarmed about. Make sure that the links to the rest of your site work and are plain HTML, then you should be indexed in good time. Having more and higher PR links to the homepage and also deep links to subpages can help make this faster, but MetropolisRobot is right, it just takes time. Googlebot has a lot to do. It'll get to you eventually.

Fresh pages on new sites with fresh links to fresh content seem to get freshbotted the most in my experience. But I've had freshbot crawl a page and then follow some links and not others, index some subpages and not others, with no apparent reason. Sometimes one fresh page will get indexed several times during a month, whereas the one right next to it on the nav bar doesn't get indexed until the update. It's weird, but you get used to it.