Is there a failsafe way to attract a number of spiders on a regular basis? I've noticed that Scooter, Pirahna, Teoma, and a few others index client's sites once and never return. Why would this be?
Also... Is there just a general *food& that can be put out on the site's contents to attract more and more spiders quickly? I've noticed that the logs have a tendancy to show one-hit-wonders and once every 2 monthers... And I want active scanning of ALL resources..
Another small issue.. Anyone know of a way to refer spiders from one site to another?
One client has 2 sites.. One of which gets some MAJOR attention... Another, gets next to nothing. Anyways to transfer these spiders back and forth? I think that it would greatly benefit the SE-findings...
Eric_Lander, is there a difference between the number of inbound links to the two sites in question? That could possibly have an influence.
You'll also want to double-check that the sites have good internal text links for spiders to follow.
I wouldn't be overly concerned about spiders other than those from the the major search engines. Some visit on a regular basis; however not all of them deep-crawl. With some you might want to make submissions of additional site pages (keep it under 5 a day from a site)to give a nudge.
skirril's point is well made about the ODP listing, which has a profound influence on your Google ranking.