Welcome to WebmasterWorld Guest from 54.144.243.34

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Robots.txt
I better get this right...
6 JVB_Mktg 4:19 am Feb 11, 2003
Robots.txt
How? Sub-Domains
2 MarkHutch 8:37 pm Feb 6, 2003
frames and robots.txt
4 snowfishin 9:15 pm Feb 3, 2003
Do proxies and Internet caches respect robots.txt?
9 Scooter24 12:06 am Feb 3, 2003
robots.txt file - what program to write it in?
I use Frontpage 2002
6 Web_Footed_Newbie 4:00 pm Jan 29, 2003
Robots.txt - What program to use to write?
I use Frontpage 2002
4 Web_Footed_Newbie 3:34 pm Jan 29, 2003
Potbot not respecting robots.txt
3 Scooter24 1:59 pm Jan 24, 2003
robots.txt and robots meta tag not mutually exclusive?
robots.txt robots meta tag
4 Check6 1:35 pm Jan 24, 2003
"User-Agent"?
...or, just User-Agent ?
13 pendanticist 3:22 am Jan 22, 2003
robots.txt command to disallow indexing
I want to only disallows robots to index a specific directory
2 wellnesscafe 4:50 am Jan 18, 2003
Quick question about robots.txt
Excluding dynamic pages
4 TheDave 4:09 am Jan 15, 2003
Why would rogue spiders obey robots.txt?
2 irock 5:29 am Jan 9, 2003
what is robots.txt file?
3 antipodes 5:57 am Jan 4, 2003
Getting ERROR_DISALLOW with Inktomi after adding new robots.txt
7 bebox 1:41 pm Dec 29, 2002
Webmasterworld's Robots.txt
A couple of questiosn for Brett
4 victor 5:00 pm Dec 27, 2002
Robots.txt Validator bug?
Christian Storm from Turnitin Robot maintainers says my robots.txt is wrong
3 scottspence 11:15 am Dec 17, 2002
How do I create a robot.txt?
9 palmpal 1:35 pm Dec 15, 2002
robots.txt to stop the BAD bots
There was a past post with the best robots.txt file?
11 Googly 11:18 pm Dec 12, 2002
Robots.txt file
disallow target_blank
3 junior 9:06 am Dec 5, 2002
robots.txt
robots.txt
3 darkroom 9:50 am Dec 3, 2002
robots.txt and mod_rewrite, meta robots, etc.
4 Trisha 4:17 am Nov 30, 2002
robot.txt syntax
7 jamesf4218 9:48 am Nov 25, 2002
Ask Jeeves robots.txt entry
what UA should be used
3 Kerrin 4:40 pm Nov 7, 2002
logic check on robots.txt
will robots.txt properly disallowing a links page prevent link pop penalty?
9 wooden 6:47 pm Nov 5, 2002
Which robots to exclude
8 biggles 1:56 am Nov 4, 2002

Featured Threads

Hot Threads This Week

Hot Threads This Month