Welcome to WebmasterWorld Guest from 54.146.248.111

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
How do I apply the robots.txt?
Seems important and need to know more!
3 askjoe 6:24 pm Mar 7, 2003
I don't want to be listed yet
Is disabling anonomas access enough?
8 currington 9:53 am Mar 5, 2003
empty robots.txt
yes or no
6 HuhuFruFru 8:32 am Mar 5, 2003
question about an empty robots.txt
5 HuhuFruFru 8:36 pm Mar 4, 2003
logs and robots.txt
7 tyrojds 7:42 pm Mar 4, 2003
Why robots.txt if you want everything indexed?
Does it even matter?
9 Larry_C 3:49 pm Mar 3, 2003
Robots.txt question
excluding every folder expect one
5 JVB_Mktg 7:54 pm Mar 2, 2003
Logs show a code 206 with robots.txt
code 206 - partial content
2 nativenewyorker 11:03 pm Mar 1, 2003
robots.txt syntax
spaces or slashes?
3 keyplyr 5:14 pm Feb 27, 2003
Strange Incident with robots.txt
error code 401
3 neh2008 4:57 pm Feb 17, 2003
robots.txt to block bots access to images
got to save bandwidth
4 stavs 3:00 pm Feb 14, 2003
ROBOTS.TXT Location
ROBOTS.TXT Location
4 neh2008 5:27 pm Feb 11, 2003
Robots.txt
I better get this right...
6 JVB_Mktg 4:19 am Feb 11, 2003
Robots.txt
How? Sub-Domains
2 MarkHutch 8:37 pm Feb 6, 2003
frames and robots.txt
4 snowfishin 9:15 pm Feb 3, 2003
Do proxies and Internet caches respect robots.txt?
9 Scooter24 12:06 am Feb 3, 2003
robots.txt file - what program to write it in?
I use Frontpage 2002
6 Web_Footed_Newbie 4:00 pm Jan 29, 2003
Robots.txt - What program to use to write?
I use Frontpage 2002
4 Web_Footed_Newbie 3:34 pm Jan 29, 2003
Potbot not respecting robots.txt
3 Scooter24 1:59 pm Jan 24, 2003
robots.txt and robots meta tag not mutually exclusive?
robots.txt robots meta tag
4 Check6 1:35 pm Jan 24, 2003
"User-Agent"?
...or, just User-Agent ?
13 pendanticist 3:22 am Jan 22, 2003
robots.txt command to disallow indexing
I want to only disallows robots to index a specific directory
2 wellnesscafe 4:50 am Jan 18, 2003
Quick question about robots.txt
Excluding dynamic pages
4 TheDave 4:09 am Jan 15, 2003
Why would rogue spiders obey robots.txt?
2 irock 5:29 am Jan 9, 2003
what is robots.txt file?
3 antipodes 5:57 am Jan 4, 2003

Featured Threads

Hot Threads This Week

Hot Threads This Month