homepage Welcome to WebmasterWorld Guest from 54.226.80.55
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt on phpbb forum
RammsteinNicCage




msg:1527601
 4:55 pm on Dec 6, 2003 (gmt 0)

Someone made a mod that allows all spiders to spider a phpBB forum. I was wondering if there would be a way to prevent them from spidering some of the sections of the forum because they aren't relevent to the site content. So, say these are the sections I have:

www.domain.com/forum/viewforum.php?f=1
www.domain.com/forum/viewforum.php?f=2
www.domain.com/forum/viewforum.php?f=3

I want the first section spidered, but not the second or third. Is this possible?

Jennifer

 

jamie




msg:1527602
 4:06 pm on Dec 12, 2003 (gmt 0)

hi jennifer,

as far as i know you can simply specify the forum you wish not to be crawled. e.g.

User-agent: *
Disallow: /phpbb/viewforum.php?f=18

this is perfectly valid and it also complies with googles robots.txt info page [google.com]

to be on the safe side though this post bumps yours to the top anyway ;-)

cheers

<added>

hi jennifer, i have just added a bit of php code to two of the phpbb files which adds a <meta name="robots" content="noindex,nofollow"> depending on the forum_id

sticky me if you wish for the code

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved