Welcome to WebmasterWorld Guest from 220.127.116.11
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
Is there such a thing as Allow: *
|4||merlin78||2:38 am May 24, 2001|
How do I do this...
|2||lizzie||8:27 pm May 19, 2001|
How effective is it anyway
|6||circuitjump||2:35 pm May 11, 2001|
|Robozilla and Robots.txt
||4||Brett_Tabke||8:11 am May 5, 2001|
|replacing robots.txt with a CGI
||7||theperlyking||7:42 pm Apr 25, 2001|
warning; robots.txt containing DOS or Mac line
|5||roma||10:35 am Apr 25, 2001|
Do you use it if you want to all all robots?
|7||WebSeeker||8:30 am Apr 16, 2001|
|Looking for complicated borderline Robots.txt
||4||Brett_Tabke||10:14 am Apr 9, 2001|
|Google Link points and Robots.txt
Google points algo and side discussion of robots.txt format ( 1 2 )
|34||tmw||2:53 pm Mar 27, 2001|
|Doorway / Hallway pages Vs. Robots.txt
Does one need to build "many" sites under one roof?
|5||Garyh||10:45 pm Mar 20, 2001|
|Yahoo and Robots.txt
||2||Brett_Tabke||9:16 pm Feb 28, 2001|
|How does robot.txt work?
||4||Acternaweb||2:59 am Feb 28, 2001|
How do I set up a Robots.txt file?
|6||CaveToad||3:57 pm Jan 31, 2001|
|Robots.txt and subdomains.
How do spiders react on robots.txt on subdomains?
|1||Rumbas||11:05 am Jan 29, 2001|
||10||graemel||10:47 pm Jan 24, 2001|
|Spiders Ignoring Robots.txt
Content Vs. Legal Link or Site Trespassing.
|6||Garyh||4:12 pm Jan 22, 2001|
|Why would I use a Robots.txt file?
||7||BoneHeadicus||12:40 am Jan 14, 2001|
|Should I be using a Robots.txt file?\n
||6||BoneHeadicus||7:40 pm Jan 10, 2001|
|To "Allow" or not to "Allow" in robots.txt
Can it force a deep crawl?
|4||webtamers||9:53 am Nov 6, 2000|
|SEs Check robots.txt and just go away?
||9||webtamers||6:14 am Nov 6, 2000|
|Aim-pro Newsletter: robots.txt and too many urls
||6||Brett_Tabke||6:22 pm Jun 29, 2000|