Forum Moderators: goodroi
Typically, pages are generated thusly:
/cat1.php?id=1
/cat1.php?id=1&page=2
/cat1.php?id=1&page=3
/cat1.php?id=1&page=4
/cat1.php?id=2
/cat1.php?id=2&page=2
/cat1.php?id=2&page=3
/cat1.php?id=2&page=4
and so on...
I want all four of the 'id=1' pages to be crawled; but I only want the first 'id=2' page crawled. Can this be done, and if so, how would that be written in a robots.txt file?