Forum Moderators: open
My site was just crawled yesterday and I just do not understand why it did only index a couple of my pages.
Every article is stored in a directory like http://www.example.com/article/1666/index.asp
and I link many articles from my homepage which is indexed in Google.
I checked the spider simulators and discovered something kind of odd.
If I use [searchengineworld.com...] I only get a couple of results for my site sixshot.com, but if I use pretty much all the others I found like http://www.netstep.org/search-engine-optimization-tools/spider-simulator.htm I get the result I want.
What am I doing wrong?
Thanks for your help...
[edited by: heini at 9:11 am (utc) on Feb. 28, 2003]
[edit reason] Sinner-G said it / thanks! [/edit]
You are not supposed to post URLs here, at least your own. I guess the spider sims URLs are ok.
I tested an ASP site and also get many more links with the netstep simulator, but a lot of those are duplicates. Did you check that with your site?
Sinner
i think your page is hard to crawl. i don't think that each article url is bad. so the spider does not find each article. even on an article page the is much overweight of navigation and boxes all around which are parsed by the spider first, so your indexed article can't be indexed that specific.
I took at look at your home page, and noticed several things -
* HTML way too complicated, spiders prefer simple. At least move all that Javascript out of the start of your HTML and into an external .js file.
* Browser compatibility problems. You don't even want to know what your site looks like in original Netscape, but it ain't pretty!
* A lot of links in the form "article.asp?id=1631" and "philboard_read.asp?id=1707". Don't do this, or at least call the param something else besides id (see other posts here about problems with that & Google)
There may be other problems as well, but I really didn't feel like wading through all that HTML/CSS/Javascript/Applets/Flash/pop-ups/pop-unders/audio/video/right-click-trapping/etc. to find out. My God, is there any "bell and whistle" technology you don't use on your home page? About the only thing I didn't see was VRML, and it's probably in there somewhere if I looked hard enough! (Reminds me of a German car, sometimes they add extra gauges to the dashboard just because it "looks cool".) No wonder some spiders have trouble with your site! Simplify!
First post eh and you certainly got a rough ride :)
Stick around it wont always be that harsh :)
One thing I did notice is that you have to log in to quite a few sections/page so that rules out search engines for one.
You might wish to consider taking some of the noise away, I found myself a little lost. Now when it looks like it might lose you from a browsers perspective you can be sure that the core content and navigation is going to be a bit of a handfull to the search engine too.
One good thing I reckon it that you have all your stuff in a DB so a major redesign for you wont be as serious as for a person who has gone that static route.
If you dont wish to go down the redesign route you could add a few site maps that point to individual pages within and see how that goes.
From my point of view there are a lot of bells and whistles that I would think of removing (flash, applets etc) These slow the page down something chronic.
Just a few things to consider.
Cheers
It is all SQL based, therefore some redesign is not that much work hard.
And at least some of the news and stuff was listed and after the last rescan I get some crappy results on google with sitesearch.
