Welcome to WebmasterWorld Guest from 220.127.116.11
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Do we need robots at all? New member here
Confused as to whether I need to add robots txt
|5||plenusvita||9:23 pm Dec 2, 2006|
|Block all spiders, but only allow well known spiders?
||2||foxfox||3:19 pm Dec 2, 2006|
|msnbot visits robots.txt and exits
||7||pemba76||12:13 pm Nov 28, 2006|
|Upload of incorrect robots.txt
What a screw up - any permanent damage?
|4||TinkyWinky||10:12 am Nov 28, 2006|
|Does excluding Googlebot create a penalty?
If other bots are allowed, is this a red flag to Google
|2||AndyA||9:26 pm Nov 27, 2006|
|what to put into robots.txt?
||11||Hitman3266||5:09 pm Nov 27, 2006|
|How long does it take for SEs to learn about a robots'txt
||4||bouncybunny||12:04 pm Nov 20, 2006|
|Yahoo! Slurp Now Supports Wildcards in robots.txt
||17||OldWolf||2:54 am Nov 20, 2006|
|Robots.txt and password-protection
||3||hwy2hel||10:56 pm Nov 16, 2006|
|Yahoo robots txt
nedd to disallow all urls with?
|2||cheaperholidays||4:54 pm Nov 16, 2006|
|robots.txt? OR .htaccess?
disallowing or protecting directories
|2||johnjoyce||4:27 pm Nov 16, 2006|
|Wildcards in robots.txt for forum software
||6||Asia_Expat||1:40 am Nov 14, 2006|
'$' and '?'
|2||sssweb||8:40 pm Nov 11, 2006|
||2||miracle||7:16 pm Nov 8, 2006|
|Link Checker / Site map generator
One that observes robots.txt and meta tags
|5||fish_eye||5:33 am Nov 7, 2006|
|Disallow by ISP
Where to get a list of ISPs and the countries they are in?
|1||youfoundjake||7:13 pm Nov 3, 2006|
|Made the mistake of putting sub folders I want allowed
disallow everthing but?
|2||Sunnyvale||7:07 pm Oct 31, 2006|
|Google remove url tool for supplemental urls a waste of time?
Should I bother removing duplicate supplementals?
|3||Tonerman||4:20 pm Oct 30, 2006|
|allow major engines, block the fleas
and to add further control
|5||youfoundjake||5:16 pm Oct 26, 2006|
|Dissallowing Dynamic Content
|2||trillianjedi||6:58 pm Oct 25, 2006|
|Combining htaccess and robots.txt?
||7||Lady18||11:37 pm Oct 24, 2006|
|disallowing a 'staging server'
||2||rden17||4:11 pm Oct 24, 2006|
||2||ssgumby||9:53 pm Oct 21, 2006|
to avoid hacking
|4||idolw||12:26 pm Oct 18, 2006|
|How to deal with a robots.txt hack attack
Guidelines to mitigate the damage from an attack
|3||Whitey||10:17 pm Oct 17, 2006|