Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
||4||count_zer0||11:21 am Dec 7, 2001|
||7||Kenny B||1:02 pm Sep 25, 2001|
|Robots.txt specific query
||8||IanTurner||7:19 pm Sep 24, 2001|
|basic robots.txt question
||5||kapow||4:31 pm Aug 2, 2001|
|I am new to all of this about Spiders...Robot.txt
|8||lightwriter||12:42 am Jul 23, 2001|
Banning dodgy spiders - do they take any notice?
|4||themoff||7:35 pm Jul 13, 2001|
Does Google honour it....
|12||Rusky||5:58 am Jul 5, 2001|
|Fear and Loathing of Robots.txt
Coming Clean - Confessions of a a robo'phobic
|12||Brett_Tabke||1:23 am Jun 25, 2001|
|Bad style in robots.txt file. Why?
||5||billyblu||10:25 am Jun 1, 2001|
||4||Froggyman||9:17 am May 26, 2001|
Is there such a thing as Allow: *
|4||merlin78||2:38 am May 24, 2001|
How do I do this...
|2||lizzie||8:27 pm May 19, 2001|
How effective is it anyway
|6||circuitjump||2:35 pm May 11, 2001|
|Robozilla and Robots.txt
||4||Brett_Tabke||8:11 am May 5, 2001|
|replacing robots.txt with a CGI
||7||theperlyking||7:42 pm Apr 25, 2001|
warning; robots.txt containing DOS or Mac line
|5||roma||10:35 am Apr 25, 2001|
Do you use it if you want to all all robots?
|7||WebSeeker||8:30 am Apr 16, 2001|
|Looking for complicated borderline Robots.txt
||4||Brett_Tabke||10:14 am Apr 9, 2001|
|Google Link points and Robots.txt
Google points algo and side discussion of robots.txt format ( 1 2 )
|34||tmw||2:53 pm Mar 27, 2001|
|Doorway / Hallway pages Vs. Robots.txt
Does one need to build "many" sites under one roof?
|5||Garyh||10:45 pm Mar 20, 2001|
|Yahoo and Robots.txt
||2||Brett_Tabke||9:16 pm Feb 28, 2001|
|How does robot.txt work?
||4||Acternaweb||2:59 am Feb 28, 2001|
How do I set up a Robots.txt file?
|6||CaveToad||3:57 pm Jan 31, 2001|
|Robots.txt and subdomains.
How do spiders react on robots.txt on subdomains?
|1||Rumbas||11:05 am Jan 29, 2001|
||10||graemel||10:47 pm Jan 24, 2001|