Welcome to WebmasterWorld Guest from 54.90.204.233

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Robots.txt
Do you use it if you want to all all robots?
7 WebSeeker 8:30 am Apr 16, 2001
Looking for complicated borderline Robots.txt
4 Brett_Tabke 10:14 am Apr 9, 2001
Google Link points and Robots.txt
Google points algo and side discussion of robots.txt format[2] ( 1 2 )
34 tmw 2:53 pm Mar 27, 2001
Doorway / Hallway pages Vs. Robots.txt
Does one need to build "many" sites under one roof?
5 Garyh 10:45 pm Mar 20, 2001
Yahoo and Robots.txt
2 Brett_Tabke 9:16 pm Feb 28, 2001
How does robot.txt work?
4 Acternaweb 2:59 am Feb 28, 2001
Mr. Roboto
How do I set up a Robots.txt file?
6 CaveToad 3:57 pm Jan 31, 2001
Robots.txt and subdomains.
How do spiders react on robots.txt on subdomains?
1 Rumbas 11:05 am Jan 29, 2001
Spider.txt
10 graemel 10:47 pm Jan 24, 2001
Spiders Ignoring Robots.txt
Content Vs. Legal Link or Site Trespassing.
6 Garyh 4:12 pm Jan 22, 2001
Why would I use a Robots.txt file?
7 BoneHeadicus 12:40 am Jan 14, 2001
Should I be using a Robots.txt file?\n
6 BoneHeadicus 7:40 pm Jan 10, 2001
To "Allow" or not to "Allow" in robots.txt
Can it force a deep crawl?
4 webtamers 9:53 am Nov 6, 2000
SEs Check robots.txt and just go away?
9 webtamers 6:14 am Nov 6, 2000
Aim-pro Newsletter: robots.txt and too many urls
6 Brett_Tabke 6:22 pm Jun 29, 2000