Welcome to WebmasterWorld Guest from 50.16.112.199

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt -- block one user-agent, allow all others
2 luma 1:44 am May 23, 2002
Robot.txt file help
How to create this file
7 beasscr 6:30 pm May 16, 2002
robots.txt
Generates a 404 error in my logs
9 zeus 1:52 pm May 3, 2002
Does robots.txt override meta tag?
8 kira 7:12 pm May 2, 2002
Robots.txt validator updated.
7 Brett_Tabke 6:48 pm Apr 19, 2002
What is robots.txt syntax
22 Alec_Doggone 9:07 am Apr 19, 2002
Submitting robots.txt to google
Would google look sooner for it?
17 Yidaki 8:19 am Apr 19, 2002
Robots.txt Pattern Matching
How Well Does It Work?
5 WebGuerrilla 1:15 pm Mar 15, 2002
Robots.txt Question?
How often does Ink check for the file??
6 MarkHutch 5:11 pm Mar 11, 2002
Do I need a robots.txt file in my directory?
Google Bot - Whats Up with it?
10 jady 9:58 pm Feb 24, 2002
How important is the Robots.txt file now?
....in 2002
29 ScottM 1:16 am Jan 28, 2002
Robots.txt Question
7 mud 3:21 am Jan 17, 2002
I am writing my 1st robots.txt :) - never bothered before
Anyone know any crawlers I should add ??
3 diddlydazz 4:11 am Jan 12, 2002
MSIECrawler not obeying my robots.txt :(
any ideas ?
5 diddlydazz 3:09 am Jan 12, 2002
Robots.txt standard
4 count_zer0 11:21 am Dec 7, 2001
robots.txt
7 Kenny B 1:02 pm Sep 25, 2001
Robots.txt specific query
8 IanTurner 7:19 pm Sep 24, 2001
basic robots.txt question
5 kapow 4:31 pm Aug 2, 2001
I am new to all of this about Spiders...Robot.txt
Help Message
8 lightwriter 12:42 am Jul 23, 2001
robots.txt
Banning dodgy spiders - do they take any notice?
4 themoff 7:35 pm Jul 13, 2001
Robots.txt
Does Google honour it....
12 Rusky 5:58 am Jul 5, 2001
Fear and Loathing of Robots.txt
Coming Clean - Confessions of a a robo'phobic
12 Brett_Tabke 1:23 am Jun 25, 2001
Bad style in robots.txt file. Why?
5 billyblu 10:25 am Jun 1, 2001
Robots.txt Validator
4 Froggyman 9:17 am May 26, 2001
Robots.txt
Is there such a thing as Allow: *
4 merlin78 2:38 am May 24, 2001

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month