Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor 2014
robots.txt on phpbb forum RammsteinNicCage
Someone made a mod that allows all spiders to spider a phpBB forum. I was wondering if there would be a way to prevent them from spidering some of the sections of the forum because they aren't relevent to the site content. So, say these are the sections I have:
I want the first section spidered, but not the second or third. Is this possible?
as far as i know you can simply specify the forum you wish not to be crawled. e.g.
this is perfectly valid and it also complies with googles
robots.txt info page [ google.com]
to be on the safe side though this post bumps yours to the top anyway ;-)
hi jennifer, i have just added a bit of php code to two of the phpbb files which adds a <meta name="robots" content="noindex,nofollow"> depending on the forum_id
sticky me if you wish for the code