Only index page was crawled...

:(

[nbozic]

I worked very hard on my forum. I spent several weeks researching and implementing URL rewriting to make my forum more search engine friendly.
When I submitted the main URL about three months ago to search engines, since then only the index page was crawled and listed.
How come the rest of the pages are not getting listed, and how come some search engines don't index my forum at all?
I gave it a lot of thought and tried many things, but I'm getting frustrated :(
Any ideas?

Thanks,

NB
<Sorry, no personal URLs. See TOS [webmasterworld.com]>

[edited by: tedster at 4:14 pm (utc) on April 17, 2004]

[PeteM]

I had a similar problem but with phpBB. Have you tried the Invision help forums?

[edited by: tedster at 4:15 pm (utc) on April 17, 2004]

[Marcia]

It's always a good idea to check and validate the robots.txt whenever there's any sign of a problem.

[nbozic]

The contents of my robots.txt file are as follows:
User-agent: *
Disallow: /forum/

Basically, I don't allow any robots to index or follow into the forum folder. Actually, all of my forum's content is in the forum folder. This should not be a problem though...
All the pages that I want indexed by search engines, I rewrote their URLs. For example, if the old URL is this:
example.com/forum/index.php?showtopic=42
I rewrote it into this:
example.com/Music_Equipment-c2/How_to_Buy_Equipment-f4/t42.html

You can see that although the page is actually in the forum folder, search engines (or anybody else) sees it as being in different folders. There is no reason I see why the category pages and topic pages are not getting indexed by search engines - the main page (index.html) is the only page that ever got indexed.

NB

[edited by: tedster at 6:58 pm (utc) on April 17, 2004]
[edit reason] use example.com [/edit]

[tedster]

Do you have good, basic HTML links to those URLs from your index page? No fancy dynamic anything at all going on, just solid HTML anchor tags with href attributes that are a valid URL.

[nbozic]

I don't have any links to specific topics (actual threads) on my index page... but no forums do that I know of. It has to be hierarchical.
I have the following forum structure:

1. Index page
2. Forum
3. Topic (thread)

It's the same structure as webmasterworld forum or any other forum, but I bet you guys are getting spidered effectively. For some reason my pages are not getting spidered.

NB

[nbozic]

I just found something very odd by searching google, it was the first result when I searched for my forum's name:
--------------------------------------------------
Only index page was crawled...
... I gave it a lot of thought and tried many things, but I'm getting frustrated :( The
URL is ****xxxxxxxx.com if you need to take a look. Any ideas? Thanks,. ...
www.webmasterworld.com/forum5/4341.htm - Similar pages
--------------------------------------------------
The page was spidered before the link was edited (sorry about that)...
but the point I'm trying to make is that my forum's index page ranked last (3rd place out of three records found). How can the actual page that you're searching for rank last, and the pages that just mention its name rank higher.
I find it very odd, and was wondering if you have any clues why. Is it link popularity? My forum is quite new and I don't think that even one site on the internet links to me. Do I have to start asking other techno music webmasters to exchange links with me?

NB

[PeteM]

You are right, I checked your site before the URL was removed. There are no backlinks to your site (some would help) and your site currently has a PR (Page Rank) of 0.