Welcome to WebmasterWorld Guest from 54.163.35.238

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
the wrong way to do robots.txt
kinda funny, whitehouse.gov robots file
13 amznVibe 7:42 pm Nov 10, 2003
Why bother
Is it worth blocking spambots with robots.txt
4 WyrdoManx 11:52 pm Nov 8, 2003
Does content type matter?
is text/html okay
2 kevinsl 8:27 am Nov 8, 2003
IE "Synchronize Offline while idle" is killing me
Synchronize Offline
3 Helice 12:46 am Nov 6, 2003
Bots ignoring my robots.txt
In place for a week, validates, but no use
12 roddy 7:10 am Nov 5, 2003
Blocking a spider from a directory
3 JVB_Mktg 4:30 am Nov 5, 2003
Robots.txt
To use... or not to use? That is the question
6 panic 1:29 am Nov 5, 2003
Robots.txt Validator
We're sorry, this robots.txt does NOT validate.
2 midge 12:48 pm Nov 3, 2003
Robot.txt - What is it, and why use it
4 laffe 4:36 am Nov 2, 2003
robots.txt syntex
How to block a directory.
6 jdancing 7:52 pm Oct 31, 2003
Robots.txt during development
Anyone advise on this?
2 stuntdubl 3:30 am Oct 30, 2003
bad Bots
3 cgj5204 4:40 am Oct 29, 2003
Restrict Specific Domain Names from Being Indexed
Using the robots.txt file on Google to restrict certain domains...
3 jmh1970 1:36 am Oct 29, 2003
New Forum : Robots.txt
1 Brett_Tabke 9:24 pm Oct 28, 2003
Put your robots.txt on a diet
Three easy ways to reduce bloat in robots.txt
20 jdMorgan 8:36 pm Oct 28, 2003
Blocking image file indexing without robots.txt
Can this be done?
2 Winedark 8:08 pm Oct 28, 2003
Robots.txt size
How long can a robots.txt be?
4 WyrdoManx 9:47 pm Oct 22, 2003
Robots.txt - Am I Missing Somthing?
Query strings and general syntax is suspected...
13 Nick_W 3:40 pm Oct 21, 2003
How to correctly set robots.txt for a subdomain?
please help
5 PatrickDeese 6:36 pm Oct 15, 2003
Robots.txt for Query Strings
Can I stop bots from following links with query stings
10 Nick_W 6:08 pm Oct 15, 2003
DMOZ Robots.txt
Crawling no longer allowed
12 Yidaki 5:38 pm Oct 14, 2003
Robots.txt Question :)
Question regarding how to use robots txt
8 Prana 9:33 pm Sep 27, 2003
robots.txt
3 domains in root
9 Alternative_Future 11:33 pm Sep 23, 2003
Robots.txt disallow: /index.php?
then /index.php?param=example still allowed?
5 Yidaki 12:17 pm Sep 21, 2003
Robots.txt exclusion question
Robots.txt exclude all but one directory?
4 olof_bailey 8:52 pm Sep 20, 2003

Featured Threads

Hot Threads This Week

Hot Threads This Month