Welcome to WebmasterWorld Guest from 54.166.33.25

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Removing Pages - Does Robots.txt Help?
3 akreider 4:17 am Dec 16, 2006
Allowing only some pages for ia archive
5 sosoo 8:20 am Dec 13, 2006
Mediapartners-Google fellow the robots.txt?
2 AlphaWu 1:41 pm Dec 12, 2006
my robots file
need help
3 experienced 12:26 am Dec 12, 2006
Disagreement between Google and a validator
Robots.txt Checker says it's so, Google robots anayler says the other way
6 Gusgsm 9:49 am Dec 11, 2006
Googlebot tripping bad bot script
5 RobertRogers 5:29 pm Dec 10, 2006
Is this bad robots.txt
1 scoobydooaustralia 8:02 pm Dec 8, 2006
duplicate pages .aspx?id=
Not sure what to do next
2 schatzm 9:01 pm Dec 6, 2006
Is this a bot?
Unwanted visitor to my website
2 brancook 8:57 pm Dec 6, 2006
Two domains mirror each other
Is it possible to a command to a robots.txt file to apply to just one?
4 Jeremy_H 9:08 pm Dec 5, 2006
Do I need to disallow files that aren't linked to?
for example, 'included' PHP & javascript pages
9 sssweb 3:34 pm Dec 4, 2006
removing permanent redirect - honoring content in page
removing permanent redirect
2 thu_household 2:56 pm Dec 4, 2006
Do we need robots at all? New member here
Confused as to whether I need to add robots txt
5 plenusvita 9:23 pm Dec 2, 2006
Block all spiders, but only allow well known spiders?
2 foxfox 3:19 pm Dec 2, 2006
msnbot visits robots.txt and exits
7 pemba76 12:13 pm Nov 28, 2006
Upload of incorrect robots.txt
What a screw up - any permanent damage?
4 TinkyWinky 10:12 am Nov 28, 2006
Does excluding Googlebot create a penalty?
If other bots are allowed, is this a red flag to Google
2 AndyA 9:26 pm Nov 27, 2006
what to put into robots.txt?
11 Hitman3266 5:09 pm Nov 27, 2006
How long does it take for SEs to learn about a robots'txt
4 bouncybunny 12:04 pm Nov 20, 2006
Yahoo! Slurp Now Supports Wildcards in robots.txt
17 OldWolf 2:54 am Nov 20, 2006
Robots.txt and password-protection
3 hwy2hel 10:56 pm Nov 16, 2006
Yahoo robots txt
nedd to disallow all urls with?
2 cheaperholidays 4:54 pm Nov 16, 2006
robots.txt? OR .htaccess?
disallowing or protecting directories
2 johnjoyce 4:27 pm Nov 16, 2006
Wildcards in robots.txt for forum software
6 Asia_Expat 1:40 am Nov 14, 2006
robots.txt syntax
'$' and '?'
2 sssweb 8:40 pm Nov 11, 2006

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month