Welcome to WebmasterWorld Guest from 54.163.210.170

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Robots.txt file
How do I do this...
2 lizzie 8:27 pm May 19, 2001
Robots.txt
How effective is it anyway
6 circuitjump 2:35 pm May 11, 2001
Robozilla and Robots.txt
4 Brett_Tabke 8:11 am May 5, 2001
replacing robots.txt with a CGI
7 theperlyking 7:42 pm Apr 25, 2001
robots.txt
warning; robots.txt containing DOS or Mac line
5 roma 10:35 am Apr 25, 2001
Robots.txt
Do you use it if you want to all all robots?
7 WebSeeker 8:30 am Apr 16, 2001
Looking for complicated borderline Robots.txt
4 Brett_Tabke 10:14 am Apr 9, 2001
Google Link points and Robots.txt
Google points algo and side discussion of robots.txt format[2] ( 1 2 )
34 tmw 2:53 pm Mar 27, 2001
Doorway / Hallway pages Vs. Robots.txt
Does one need to build "many" sites under one roof?
5 Garyh 10:45 pm Mar 20, 2001
Yahoo and Robots.txt
2 Brett_Tabke 9:16 pm Feb 28, 2001
How does robot.txt work?
4 Acternaweb 2:59 am Feb 28, 2001
Mr. Roboto
How do I set up a Robots.txt file?
6 CaveToad 3:57 pm Jan 31, 2001
Robots.txt and subdomains.
How do spiders react on robots.txt on subdomains?
1 Rumbas 11:05 am Jan 29, 2001
Spider.txt
10 graemel 10:47 pm Jan 24, 2001
Spiders Ignoring Robots.txt
Content Vs. Legal Link or Site Trespassing.
6 Garyh 4:12 pm Jan 22, 2001
Why would I use a Robots.txt file?
7 BoneHeadicus 12:40 am Jan 14, 2001
Should I be using a Robots.txt file?\n
6 BoneHeadicus 7:40 pm Jan 10, 2001
To "Allow" or not to "Allow" in robots.txt
Can it force a deep crawl?
4 webtamers 9:53 am Nov 6, 2000
SEs Check robots.txt and just go away?
9 webtamers 6:14 am Nov 6, 2000
Aim-pro Newsletter: robots.txt and too many urls
6 Brett_Tabke 6:22 pm Jun 29, 2000

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members