Welcome to WebmasterWorld Guest from 54.167.155.147

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Hiding robots.txt
21 keyplyr 1:45 am Dec 2, 2003
Did you re-test?
3 hybridx 2:51 pm Nov 28, 2003
Weird Code Return from Googlebot when robots.txt was spidered
3 5x54u 9:14 pm Nov 25, 2003
Problem with robots.txt
4 Nisse 8:08 am Nov 25, 2003
robots and dynamic url
How to ban robots from a dynamic url
3 Ledfish 8:09 pm Nov 23, 2003
Googlebot Wildcard
5 payday 11:26 am Nov 22, 2003
Importance of Hotbot
5 slobizman 9:08 am Nov 20, 2003
How long until robots.txt works?
11 slobizman 3:23 pm Nov 18, 2003
Unknown robots
crawl, spider, and robot.
2 synergy 4:10 am Nov 16, 2003
Would crawlers crawl this?
Usiing the correct robots.txt technique
4 ulysee 3:02 pm Nov 15, 2003
internetseer
billing for requested data
14 plumsauce 7:47 pm Nov 14, 2003
Pros and cons of having
a robots.txt file?
3 ideavirus 5:32 pm Nov 13, 2003
Does anyone know what this is?
I am thinking virus infected machine but unsure
3 Crowbar 4:28 pm Nov 12, 2003
Blocking .log file with robots.txt
Is it necessary, and syntax...?
7 Robert_Charlton 6:59 am Nov 11, 2003
How to make a page *inaccessible* to searches
without using password protection?
12 halfandhalf 8:04 pm Nov 10, 2003
the wrong way to do robots.txt
kinda funny, whitehouse.gov robots file
13 amznVibe 7:42 pm Nov 10, 2003
Why bother
Is it worth blocking spambots with robots.txt
4 WyrdoManx 11:52 pm Nov 8, 2003
Does content type matter?
is text/html okay
2 kevinsl 8:27 am Nov 8, 2003
IE "Synchronize Offline while idle" is killing me
Synchronize Offline
3 Helice 12:46 am Nov 6, 2003
Bots ignoring my robots.txt
In place for a week, validates, but no use
12 roddy 7:10 am Nov 5, 2003
Blocking a spider from a directory
3 JVB_Mktg 4:30 am Nov 5, 2003
Robots.txt
To use... or not to use? That is the question
6 panic 1:29 am Nov 5, 2003
Robots.txt Validator
We're sorry, this robots.txt does NOT validate.
2 midge 12:48 pm Nov 3, 2003
Robot.txt - What is it, and why use it
4 laffe 4:36 am Nov 2, 2003
robots.txt syntex
How to block a directory.
6 jdancing 7:52 pm Oct 31, 2003