I have inherited resposibilites of a dynamically driven site. It was not set up with robots.txt in mind and changes cannot be made to correct that. Typically, pages are generated thusly:
/cat1.php?id=1
/cat1.php?id=1&page=2
/cat1.php?id=1&page=3
/cat1.php?id=1&page=4
/cat1.php?id=2
/cat1.php?id=2&page=2
/cat1.php?id=2&page=3
/cat1.php?id=2&page=4
and so on...
I want all four of the 'id=1' pages to be crawled; but I only want the first 'id=2' page crawled. Can this be done, and if so, how would that be written in a robots.txt file?
