w3c linkchecker crawls my dynamic pages

[shasan]

I just used mod_rewrite to convert some URL's on my site to 'search engine friendly' URL's i.e:

[mydomain.com...]

can be reached through

www.mydomain.com/index/category/2/id/5

then I went to w3c's link checker and pointed it to www.mydomain.com and checked off the 'check recursively' option, which basically amounts to a crawl of my website.

I still have some *.php?paramenter=blah kind of pages on the site, and they get pretty long. W3C link checker is crawling these with no problem whatsoever.

So what's the big deal about mod_rewrite search engine friendly URL's... I've read so much about this, but if w3c can crawl the pages, how is it possible that SE's can't?

curious.

[closed]

They can. But if the links and the addresses of the pages are being created dynamically, for example, if a tracking id were used as a parameter, the list of pages to be visited on any given site that uses dynamic pages could never end, leading to a waste of time and bandwidth for both you and the search engine team. I'm sure that isn't the only reason why some crawlers may not follow dynamic links, but that's what I've got for right now.

It may help to read this:
[webmasterworld.com ]

You should also keep in mind that link checkers usually just check the status of pages (e.g., not found, redirected, etc.), while SEs do other things as well, such as examining the content of pages.