Hi! Can anyone out there help me with spider indexing techniques...

Spidering Indexing Techniques

[sujannandy]

Hello friends,
Can anyone out there help me explain how spiders or robots or crawlers or whatever you name them ... indexes a web page. Do they employ specific movement techniques like left ... bottom ... right ... top ... or do they look for color codes like "blues" , "blacks" etc. Do they have any algorithms defined within to seek out unethical doorway pages like pages with full text ¦ pages < 10 kb with 1 or 2 links ¦ or pages with refresh tags and url linking to actual pages?

Please do reply with examples if possible!
Sujan ;-D

[edited by: pageoneresults at 1:56 pm (utc) on June 16, 2003]
[edit reason] Removed email address. [/edit]

[trillianjedi]

Welcome to WebmasterWorld Sujan!

I'm afraid I can't answer your (interesting) question, but I can bump the thread for you... ;)

You should remove your email addy from your post, putting emails and URL's of any kind into threads is against this sites Terms of Service.

TJ

[Oaf357]

Spiders, crawler, bots, etc. typically crawl straight down your page. When I say straight down I mean open your page in your favorite editor, the bot starts at the top and goes down to the bottom.

Most bots from the major SEs employ algorithms to pick up things like keyword density, spam techniques, etc. I don't know specifics about which bot does what but the general SEO guidelines detailed in every forum on WW do apply.

I would highly recommend reading up in all the SE related forums to determine how those search engine's crawlers react to certain content/information.

Welcome to WebmasterWorld!