Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Disallowing Indexed files in Google
||5||leo11877||2:01 pm Dec 23, 2006|
|Disallowing by date.
Is it possible to keep some page and disallow new ones...?
|3||bouncybunny||8:31 am Dec 20, 2006|
|Worried about profile links
||4||Asia_Expat||3:53 pm Dec 19, 2006|
|Is it possible to exclude a URL by specifying an anchor?
|3||bouncybunny||12:40 am Dec 17, 2006|
|Removing Pages - Does Robots.txt Help?
||3||akreider||4:17 am Dec 16, 2006|
|Allowing only some pages for ia archive
||5||sosoo||8:20 am Dec 13, 2006|
|Mediapartners-Google fellow the robots.txt?
||2||AlphaWu||1:41 pm Dec 12, 2006|
|my robots file
|3||experienced||12:26 am Dec 12, 2006|
|Disagreement between Google and a validator
Robots.txt Checker says it's so, Google robots anayler says the other way
|6||Gusgsm||9:49 am Dec 11, 2006|
|Googlebot tripping bad bot script
||5||RobertRogers||5:29 pm Dec 10, 2006|
|Is this bad robots.txt
||1||scoobydooaustralia||8:02 pm Dec 8, 2006|
|duplicate pages .aspx?id=
Not sure what to do next
|2||schatzm||9:01 pm Dec 6, 2006|
|Is this a bot?
Unwanted visitor to my website
|2||brancook||8:57 pm Dec 6, 2006|
|Two domains mirror each other
Is it possible to a command to a robots.txt file to apply to just one?
|4||Jeremy_H||9:08 pm Dec 5, 2006|
|Do I need to disallow files that aren't linked to?
|9||sssweb||3:34 pm Dec 4, 2006|
|removing permanent redirect - honoring content in page
removing permanent redirect
|2||thu_household||2:56 pm Dec 4, 2006|
|Do we need robots at all? New member here
Confused as to whether I need to add robots txt
|5||plenusvita||9:23 pm Dec 2, 2006|
|Block all spiders, but only allow well known spiders?
||2||foxfox||3:19 pm Dec 2, 2006|
|msnbot visits robots.txt and exits
||7||pemba76||12:13 pm Nov 28, 2006|
|Upload of incorrect robots.txt
What a screw up - any permanent damage?
|4||TinkyWinky||10:12 am Nov 28, 2006|
|Does excluding Googlebot create a penalty?
If other bots are allowed, is this a red flag to Google
|2||AndyA||9:26 pm Nov 27, 2006|
|what to put into robots.txt?
||11||Hitman3266||5:09 pm Nov 27, 2006|
|How long does it take for SEs to learn about a robots'txt
||4||bouncybunny||12:04 pm Nov 20, 2006|
|Yahoo! Slurp Now Supports Wildcards in robots.txt
||17||OldWolf||2:54 am Nov 20, 2006|
|Robots.txt and password-protection
||3||hwy2hel||10:56 pm Nov 16, 2006|