I always have a nagging doubt when I build a dynamic web site. I go after the functionality and then wonder if the site is getting spidered correctly (or at all). Does anyone have some tips or tools that would allow me to see what a potential Spider might see - e.g. to test these dynamic creations? Or if you don't use any special tools, how do you evaluate your success with what's readily available (Google PR, SE Result sets, etc...)?
I know I can review the stats logs but this seems almost impossible to relate to. What I mean is that with dynamic content in a page called content.php, how could I know what the Spider saw? In one case, I have a single page that handles all calls to the archives. It determines the content category and adjusts the page navigation accordingly then presents the content. I can tell the page has been read XXX times with the log info but nothing more (I hope you'll correct me if I'm wrong). What I want to know (without having to build the tool to do it) is what the Spider saw when it viewed that page and if it was even legible to the Spider.
