Welcome to WebmasterWorld Guest from 54.163.35.238

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and dynamic pages

     

seasalt

7:58 pm on Jan 5, 2010 (gmt 0)

10+ Year Member



I have inherited resposibilites of a dynamically driven site. It was not set up with robots.txt in mind and changes cannot be made to correct that.

Typically, pages are generated thusly:

/cat1.php?id=1
/cat1.php?id=1&page=2
/cat1.php?id=1&page=3
/cat1.php?id=1&page=4

/cat1.php?id=2
/cat1.php?id=2&page=2
/cat1.php?id=2&page=3
/cat1.php?id=2&page=4

and so on...

I want all four of the 'id=1' pages to be crawled; but I only want the first 'id=2' page crawled. Can this be done, and if so, how would that be written in a robots.txt file?

goodroi

1:52 pm on Jan 7, 2010 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The answer to your question is yes it can be done. If you research the numerous posts already covering this here on WebmasterWorld you will learn how to write your own robots.txt.

ps it is nice to say please and thank you when you are asking people to help you write your robots.txt

neostar

8:03 am on Feb 14, 2010 (gmt 0)

5+ Year Member



Yeah, it can be done, actually its prety easy for that. Just do what goodroid said.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month