Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|robots.txt -- block one user-agent, allow all others
||2||luma||1:44 am May 23, 2002|
|Robot.txt file help
How to create this file
|7||beasscr||6:30 pm May 16, 2002|
Generates a 404 error in my logs
|9||zeus||1:52 pm May 3, 2002|
|Does robots.txt override meta tag?
||8||kira||7:12 pm May 2, 2002|
|Robots.txt validator updated.
||7||Brett_Tabke||6:48 pm Apr 19, 2002|
|What is robots.txt syntax
||22||Alec_Doggone||9:07 am Apr 19, 2002|
|Submitting robots.txt to google
Would google look sooner for it?
|17||Yidaki||8:19 am Apr 19, 2002|
|Robots.txt Pattern Matching
How Well Does It Work?
|5||WebGuerrilla||1:15 pm Mar 15, 2002|
How often does Ink check for the file??
|6||MarkHutch||5:11 pm Mar 11, 2002|
|Do I need a robots.txt file in my directory?
Google Bot - Whats Up with it?
|10||jady||9:58 pm Feb 24, 2002|
|How important is the Robots.txt file now?
|29||ScottM||1:16 am Jan 28, 2002|
||7||mud||3:21 am Jan 17, 2002|
|I am writing my 1st robots.txt :) - never bothered before
Anyone know any crawlers I should add ??
|3||diddlydazz||4:11 am Jan 12, 2002|
|MSIECrawler not obeying my robots.txt :(
any ideas ?
|5||diddlydazz||3:09 am Jan 12, 2002|
||4||count_zer0||11:21 am Dec 7, 2001|
||7||Kenny B||1:02 pm Sep 25, 2001|
|Robots.txt specific query
||8||IanTurner||7:19 pm Sep 24, 2001|
|basic robots.txt question
||5||kapow||4:31 pm Aug 2, 2001|
|I am new to all of this about Spiders...Robot.txt
|8||lightwriter||12:42 am Jul 23, 2001|
Banning dodgy spiders - do they take any notice?
|4||themoff||7:35 pm Jul 13, 2001|
Does Google honour it....
|12||Rusky||5:58 am Jul 5, 2001|
|Fear and Loathing of Robots.txt
Coming Clean - Confessions of a a robo'phobic
|12||Brett_Tabke||1:23 am Jun 25, 2001|
|Bad style in robots.txt file. Why?
||5||billyblu||10:25 am Jun 1, 2001|
||4||Froggyman||9:17 am May 26, 2001|
Is there such a thing as Allow: *
|4||merlin78||2:38 am May 24, 2001|