Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Where Can A Newbie Find More Info?

I need to learn more...



8:28 pm on Jan 23, 2001 (gmt 0)

Inactive Member
Account Expired


I've noticed that Searchengineworld.com and webmasterworld.com are great places to go to learn about Search Engine Optimization, but where does a newbie go to learn about Search Engine Spider Identification besides this forum. I've built a great website (in my opinion) and now I've submitted to enough search engines that I'm finally getting hits on my website. However, I've got many questions. For example, my statistics software (using LiveStats 5.0 software from MediaHouse) shows that I am getting about 5000 sessions a week, but only about 20% of those are referred sessions. Is the other 80% coming from Spiders? I've also noticed that a lot of my stats show stuff like "cache-ra04.proxy.aol.com". These are filling up all over the place and I don't know why they're so popular. Can anyone help or point me to the right direction so I can find out more information? Thank-you so much.


6:56 am on Jan 24, 2001 (gmt 0)

Full Member

joined:Jan 24, 2001
votes: 0

Try the Web Robots Database at:


It's a little outdated but still has a lot of good info and some good links.

6:20 pm on Jan 24, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 12, 2000
votes: 0

Here are more spider links to check out:

[edit: url snipped: bt]



List of resources:
Intelligent Agents WWW sites

6:37 pm on Jan 24, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 13

snipped the urls since jk has dropped urls many times this week. nuff.
11:07 pm on Jan 24, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 16, 2004
votes: 0

Hey cool! WebRookie, you are officially the first person to actually recommend my site, and not as an example of just how "Under Construction" a site can be. :)