Welcome to WebmasterWorld Guest from 23.22.17.192

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt to block bots access to images
got to save bandwidth
4 stavs 3:00 pm Feb 14, 2003
ROBOTS.TXT Location
ROBOTS.TXT Location
4 neh2008 5:27 pm Feb 11, 2003
Robots.txt
I better get this right...
6 JVB_Mktg 4:19 am Feb 11, 2003
Robots.txt
How? Sub-Domains
2 MarkHutch 8:37 pm Feb 6, 2003
frames and robots.txt
4 snowfishin 9:15 pm Feb 3, 2003
Do proxies and Internet caches respect robots.txt?
9 Scooter24 12:06 am Feb 3, 2003
robots.txt file - what program to write it in?
I use Frontpage 2002
6 Web_Footed_Newbie 4:00 pm Jan 29, 2003
Robots.txt - What program to use to write?
I use Frontpage 2002
4 Web_Footed_Newbie 3:34 pm Jan 29, 2003
Potbot not respecting robots.txt
3 Scooter24 1:59 pm Jan 24, 2003
robots.txt and robots meta tag not mutually exclusive?
robots.txt robots meta tag
4 Check6 1:35 pm Jan 24, 2003
"User-Agent"?
...or, just User-Agent ?
13 pendanticist 3:22 am Jan 22, 2003
robots.txt command to disallow indexing
I want to only disallows robots to index a specific directory
2 wellnesscafe 4:50 am Jan 18, 2003
Quick question about robots.txt
Excluding dynamic pages
4 TheDave 4:09 am Jan 15, 2003
Why would rogue spiders obey robots.txt?
2 irock 5:29 am Jan 9, 2003
what is robots.txt file?
3 antipodes 5:57 am Jan 4, 2003
Getting ERROR_DISALLOW with Inktomi after adding new robots.txt
7 bebox 1:41 pm Dec 29, 2002
Webmasterworld's Robots.txt
A couple of questiosn for Brett
4 victor 5:00 pm Dec 27, 2002
Robots.txt Validator bug?
Christian Storm from Turnitin Robot maintainers says my robots.txt is wrong
3 scottspence 11:15 am Dec 17, 2002
How do I create a robot.txt?
9 palmpal 1:35 pm Dec 15, 2002
robots.txt to stop the BAD bots
There was a past post with the best robots.txt file?
11 Googly 11:18 pm Dec 12, 2002
Robots.txt file
disallow target_blank
3 junior 9:06 am Dec 5, 2002
robots.txt
robots.txt
3 darkroom 9:50 am Dec 3, 2002
robots.txt and mod_rewrite, meta robots, etc.
4 Trisha 4:17 am Nov 30, 2002
robot.txt syntax
7 jamesf4218 9:48 am Nov 25, 2002
Ask Jeeves robots.txt entry
what UA should be used
3 Kerrin 4:40 pm Nov 7, 2002