Welcome to WebmasterWorld Guest from 54.162.152.232

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
How to disallow doorway pages from indexing?
4 Voyteck 2:30 pm Mar 1, 2004
Directory patterns in query strings
Will robots.txt block them?
2 Wail 8:13 pm Feb 29, 2004
list of bots to ban
it's getting ridiculous but don't want to ban any useful ones
2 walkman 4:48 pm Feb 29, 2004
should the file .htaccess visible
7 newbies 3:44 am Feb 28, 2004
Can I disallow an entire directory but one file?
too many files to list, plus I'd be advertising them
6 walkman 4:13 am Feb 26, 2004
Robots.txt and Index,nofollow
do they have to match
4 graywolf 10:38 pm Feb 22, 2004
New to Robots
New to Robots
7 gotmallz 8:32 pm Feb 22, 2004
hmmm confused
this makes no sense
1 gotmallz 6:15 am Feb 22, 2004
Confirm I've done this right please
9 Essel 5:11 pm Feb 19, 2004
Banning from the entire webserver?
2 Essel 5:57 pm Feb 18, 2004
Robots.txt Error
2 Takshaka 5:24 pm Feb 18, 2004
Robots.txt and SERPS
Blocking serps with robots.txt files and more
2 MasterKosmos 6:36 pm Feb 17, 2004
Inktomi Spider Behavior
what does this mean
5 graywolf 12:31 am Feb 17, 2004
Need help making robots to block all spiders
9 budbiss 7:47 pm Feb 16, 2004
Selective bot blocking
Scanning rules
2 fabfurs 11:43 am Feb 16, 2004
Do I need to
2 ronhollin 6:46 am Feb 16, 2004
How important is having a properly configured robots?
4 budbiss 3:45 pm Feb 14, 2004
DOS Line Enders
How to download sample robots.txt files
3 webman2 10:30 am Feb 14, 2004
robots.txt and template pages
Does blocking templates prevent spiders from following links?
2 butterfi 12:59 am Feb 12, 2004
disallow is allowing
Is this done correctly?
1 DavidAtWork 6:25 pm Feb 11, 2004
I don't have a robots.txt file..error 404
Can't get good Google or Yahoo placement
6 jagbitsllc 2:17 am Feb 11, 2004
WebmasterWorldForumBot?
what is this?
2 Trisha 10:33 pm Feb 10, 2004
How to stop robots on .php files
3 djtaverner 6:40 pm Feb 10, 2004
What in the world is a robot.txt?
I need this to be stated simple for me
5 Claudia_T 6:35 pm Feb 10, 2004
Help! I need to disallow a specific query string
1 Hissingsid 11:26 am Feb 10, 2004

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members