Robot Crawls

Is there a top ten to do list?

[theleveller]

Hi,

We have google, alexa, altavista etc, is there a top ten/twenty list of robots that we can invite to crawl our sites.

Im sure there are sites that crawl for free rather than paid for inclusion but its knowing who, what, where and when.

Submitting sites to engines is all well and good but knowing that they exist in the first place would go a long way to helping a few newbies..

Cheers

[troels nybo nielsen]

You have not mentioned Inktomi, Fast and Teoma. If their robots are visiting your sites you hardly need to worry about others.

There may be robots that you would rather live without. But that's another story.

[theleveller]

I take it then it would be good practice to stick with the others mentioned, which we have done.

But with all the other directory clones coming up, would we be better leaving these to hopefully get updated from the feeds from the main sites?

To be honest im sitting in the 'wait 6 months phase', and im just eager to keep ontop of this rather than as put in the steps to SEO "submit and forget"

Looks like ill get back to content rather than searching to invite crawls...

Cheers

[troels nybo nielsen]

> get back to content

Good idea. Create (lots of) good content, make a good internal structure, get incoming links, read WebmasterWorld and let search engines index you as best they can.