Page is a not externally linkable
seasalt - 7:58 pm on Jan 5, 2010 (gmt 0)
Typically, pages are generated thusly: /cat1.php?id=1 /cat1.php?id=2 and so on... I want all four of the 'id=1' pages to be crawled; but I only want the first 'id=2' page crawled. Can this be done, and if so, how would that be written in a robots.txt file?
I have inherited resposibilites of a dynamically driven site. It was not set up with robots.txt in mind and changes cannot be made to correct that.
/cat1.php?id=1&page=2
/cat1.php?id=1&page=3
/cat1.php?id=1&page=4
/cat1.php?id=2&page=2
/cat1.php?id=2&page=3
/cat1.php?id=2&page=4