Getting Your Website Index/Crawled

[bostonseo]

I am pretty ignorant how you can control how often the search engines index/spider your pages for updates. Is this something you can control/specify? Any help greatly appreciated.

[yowza]

The more links you get, the more they spider your site. I don't think there is anything else you can do; otherwise, I, and everyone else, would tell them to come back all the time.

[Gorilla]

For Google, the PageRank seems to control how often a page is read by Googlebot. A page with PR5 is usually read every day.

[pleeker]

Continue to add fresh, quality content to your web site. It will take a while, but the spiders are able to learn where to go to find fresh content and they will come back more often. The more food you give them, the more they'll visit.

That's an over-simplification, but hopefully answers the question in general terms.

[simonuk]

It "can" take a long time for the engines to know to keep checking back. If it's a new site you have a long road ahead unless you find an active site which will link to you.

One of my sites I've been updating daily for over a year and it was after 6 months the SE's started taking notice and now I can't keep them away; my daily logs are crammed full of spider bots.

By updating I mean adding one or two new pages every day or every other day.

Simon.

[pleeker]

One of my sites I've been updating daily for over a year and it was after 6 months the SE's started taking notice and now I can't keep them away; my daily logs are crammed full of spider bots.

I've also seen things happen in about the span of 6 months. And agreed, too, on the idea of incoming links. The more incoming links, the more often the spider will visit, and that should mean the sooner the spider learns what a great job you're doing adding fresh, quality content. Good points, simonuk.

These spiders just need to be trained, and it takes time. ;) They're attracted to activity, both in the form of new content and incoming links.

[knoir]

My advice is to write a page everyday. Check your logs and see what your visitors are searching for. Create content for them. Find many websites to trade links with.

Jonathan