Welcome to WebmasterWorld Guest from 54.166.167.206

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
wildcard disallow for pages with varibles
would this work for pages ending with?keep= : Disallow: *keep=*?
2 The_Rabbit 5:25 pm Dec 8, 2003
disallowing based on part of a query string
3 jibbajabba 4:39 pm Dec 2, 2003
Whois.sc Probe/Bot
Prevent Snapshot Cache
1 CustomFit 3:41 am Dec 2, 2003
Hiding robots.txt
21 keyplyr 1:45 am Dec 2, 2003
Did you re-test?
3 hybridx 2:51 pm Nov 28, 2003
Weird Code Return from Googlebot when robots.txt was spidered
3 5x54u 9:14 pm Nov 25, 2003
Problem with robots.txt
4 Nisse 8:08 am Nov 25, 2003
robots and dynamic url
How to ban robots from a dynamic url
3 Ledfish 8:09 pm Nov 23, 2003
Googlebot Wildcard
5 payday 11:26 am Nov 22, 2003
Importance of Hotbot
5 slobizman 9:08 am Nov 20, 2003
How long until robots.txt works?
11 slobizman 3:23 pm Nov 18, 2003
Unknown robots
crawl, spider, and robot.
2 synergy 4:10 am Nov 16, 2003
Would crawlers crawl this?
Usiing the correct robots.txt technique
4 ulysee 3:02 pm Nov 15, 2003
internetseer
billing for requested data
14 plumsauce 7:47 pm Nov 14, 2003
Pros and cons of having
a robots.txt file?
3 ideavirus 5:32 pm Nov 13, 2003
Does anyone know what this is?
I am thinking virus infected machine but unsure
3 Crowbar 4:28 pm Nov 12, 2003
Blocking .log file with robots.txt
Is it necessary, and syntax...?
7 Robert_Charlton 6:59 am Nov 11, 2003
How to make a page *inaccessible* to searches
without using password protection?
12 halfandhalf 8:04 pm Nov 10, 2003
the wrong way to do robots.txt
kinda funny, whitehouse.gov robots file
13 amznVibe 7:42 pm Nov 10, 2003
Why bother
Is it worth blocking spambots with robots.txt
4 WyrdoManx 11:52 pm Nov 8, 2003
Does content type matter?
is text/html okay
2 kevinsl 8:27 am Nov 8, 2003
IE "Synchronize Offline while idle" is killing me
Synchronize Offline
3 Helice 12:46 am Nov 6, 2003
Bots ignoring my robots.txt
In place for a week, validates, but no use
12 roddy 7:10 am Nov 5, 2003
Blocking a spider from a directory
3 JVB_Mktg 4:30 am Nov 5, 2003
Robots.txt
To use... or not to use? That is the question
6 panic 1:29 am Nov 5, 2003

Featured Threads

Hot Threads This Week

Hot Threads This Month