homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

General Spider Attractants...
Anyone know of quick solutions?

10+ Year Member

Msg#: 639 posted 1:30 pm on May 8, 2001 (gmt 0)

Is there a failsafe way to attract a number of spiders on a regular basis? I've noticed that Scooter, Pirahna, Teoma, and a few others index client's sites once and never return. Why would this be?

Also... Is there just a general *food& that can be put out on the site's contents to attract more and more spiders quickly? I've noticed that the logs have a tendancy to show one-hit-wonders and once every 2 monthers... And I want active scanning of ALL resources..

Anyways, ideas... are much appreciated. Thanks!



10+ Year Member

Msg#: 639 posted 1:49 pm on May 8, 2001 (gmt 0)

Another small issue.. Anyone know of a way to refer spiders from one site to another?

One client has 2 sites.. One of which gets some MAJOR attention... Another, gets next to nothing. Anyways to transfer these spiders back and forth? I think that it would greatly benefit the SE-findings...

Thanks again to whomever might help~


10+ Year Member

Msg#: 639 posted 11:43 pm on May 11, 2001 (gmt 0)

There are a number of way to atract spiders, some of them free, some of them less so:

a) Submit the url in question to all SE's you know (free submit is enough, usually).

b) Thrive to get an entry in the ODP (open directory project). While you wont see many recognisable referrals from dmoz.org, quite a few "minor" search engines use the ODP entries, or part of them.

c) See that your pages possess meaningful meta TAGS. Some spiders will only include those meta tags

Referral of the spider to other pages/sites usually is done by anchors (a href="..")

Other than those I dont know any generic tips I could give, different spiders look for different things.


WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 639 posted 8:20 am on May 12, 2001 (gmt 0)

Eric_Lander, is there a difference between the number of inbound links to the two sites in question? That could possibly have an influence.

You'll also want to double-check that the sites have good internal text links for spiders to follow.

I wouldn't be overly concerned about spiders other than those from the the major search engines. Some visit on a regular basis; however not all of them deep-crawl. With some you might want to make submissions of additional site pages (keep it under 5 a day from a site)to give a nudge.

skirril's point is well made about the ODP listing, which has a profound influence on your Google ranking.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved