homepage Welcome to WebmasterWorld Guest from 54.227.34.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt and dynamic pages
seasalt

10+ Year Member



 
Msg#: 4054706 posted 7:58 pm on Jan 5, 2010 (gmt 0)

I have inherited resposibilites of a dynamically driven site. It was not set up with robots.txt in mind and changes cannot be made to correct that.

Typically, pages are generated thusly:

/cat1.php?id=1
/cat1.php?id=1&page=2
/cat1.php?id=1&page=3
/cat1.php?id=1&page=4

/cat1.php?id=2
/cat1.php?id=2&page=2
/cat1.php?id=2&page=3
/cat1.php?id=2&page=4

and so on...

I want all four of the 'id=1' pages to be crawled; but I only want the first 'id=2' page crawled. Can this be done, and if so, how would that be written in a robots.txt file?

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4054706 posted 1:52 pm on Jan 7, 2010 (gmt 0)

The answer to your question is yes it can be done. If you research the numerous posts already covering this here on WebmasterWorld you will learn how to write your own robots.txt.

ps it is nice to say please and thank you when you are asking people to help you write your robots.txt

neostar



 
Msg#: 4054706 posted 8:03 am on Feb 14, 2010 (gmt 0)

Yeah, it can be done, actually its prety easy for that. Just do what goodroid said.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved