I run a pretty big forum for musicians (invision board software). Lately, my links have dropped out of the Google index. My wiki gets spidered and indexed fine, but my forum doesn't get indexed anymore. I am confident this is a problem on my part, and I will explain why. I think I have figured out the problem, but I don't know how to solve it:
The search engines are only spidering and indexing my session id's. That's it. How do I know? Luckily, through a control. I used Swish-e (a software site search) to spider my forum about 4 months ago. It spidered everything (session id's, regular urls, etc) However, recently, Swish-e will only spider session id's. If I block the session id's from being spidered (in robots.txt), swish-e will NOT spider and index any part of my forum. Note, swish-e will spider the rest of my site fine.
On another site search software (called orca search) the only way it will spider my forum is if I allow cookies. It will spider the rest of my site fine without the need to allow cookies.
Therefore, I conclude Google is behaving in a similar way as the above spiders. Thus, the reason why my forum is now out of the Google index.
Please help, my site is stagnant because of this problem, and I don't know how to fix it. My site has a pagerank 7, and has been online since 1997 with a great reputation. I don't see any reason for Google to all of a sudden do this. For the reasons I mentioned, I believe this is a technical problem on my side, and not Google's fault. I am guessing this is a cookie problem, but I don't know what to do and I have no idea what I did to create this error. Thanks for taking the time to read.