Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
||7||mud||3:21 am Jan 17, 2002|
|I am writing my 1st robots.txt :) - never bothered before
Anyone know any crawlers I should add ??
|3||diddlydazz||4:11 am Jan 12, 2002|
|MSIECrawler not obeying my robots.txt :(
any ideas ?
|5||diddlydazz||3:09 am Jan 12, 2002|
||4||count_zer0||11:21 am Dec 7, 2001|
||7||Kenny B||1:02 pm Sep 25, 2001|
|Robots.txt specific query
||8||IanTurner||7:19 pm Sep 24, 2001|
|basic robots.txt question
||5||kapow||4:31 pm Aug 2, 2001|
|I am new to all of this about Spiders...Robot.txt
|8||lightwriter||12:42 am Jul 23, 2001|
Banning dodgy spiders - do they take any notice?
|4||themoff||7:35 pm Jul 13, 2001|
Does Google honour it....
|12||Rusky||5:58 am Jul 5, 2001|
|Fear and Loathing of Robots.txt
Coming Clean - Confessions of a a robo'phobic
|12||Brett_Tabke||1:23 am Jun 25, 2001|
|Bad style in robots.txt file. Why?
||5||billyblu||10:25 am Jun 1, 2001|
||4||Froggyman||9:17 am May 26, 2001|
Is there such a thing as Allow: *
|4||merlin78||2:38 am May 24, 2001|
How do I do this...
|2||lizzie||8:27 pm May 19, 2001|
How effective is it anyway
|6||circuitjump||2:35 pm May 11, 2001|
|Robozilla and Robots.txt
||4||Brett_Tabke||8:11 am May 5, 2001|
|replacing robots.txt with a CGI
||7||theperlyking||7:42 pm Apr 25, 2001|
warning; robots.txt containing DOS or Mac line
|5||roma||10:35 am Apr 25, 2001|
Do you use it if you want to all all robots?
|7||WebSeeker||8:30 am Apr 16, 2001|
|Looking for complicated borderline Robots.txt
||4||Brett_Tabke||10:14 am Apr 9, 2001|
|Google Link points and Robots.txt
Google points algo and side discussion of robots.txt format ( 1 2 )
|34||tmw||2:53 pm Mar 27, 2001|
|Doorway / Hallway pages Vs. Robots.txt
Does one need to build "many" sites under one roof?
|5||Garyh||10:45 pm Mar 20, 2001|
|Yahoo and Robots.txt
||2||Brett_Tabke||9:16 pm Feb 28, 2001|
|How does robot.txt work?
||4||Acternaweb||2:59 am Feb 28, 2001|