Welcome to WebmasterWorld Guest from 220.127.116.11
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|How to avoid getting print-pages indexed?
What's the best way for Google?
|1||simczak||3:42 pm Jul 24, 2005|
|Robots.txt stops automated link software
||2||jdhuk||11:01 pm Jul 18, 2005|
Notepad or editor in UNIX
|3||Bali_Freak||12:15 pm Jul 18, 2005|
|selectively including folders
that are beneath a folder excluded
|2||stu2||5:20 am Jul 17, 2005|
|Would this work? Robots exclusion or considered cloaking?
||5||dvldvl3||5:11 am Jul 17, 2005|
|Previously Indexed Files-Want To Exclude Now
|3||buview1||12:23 pm Jul 16, 2005|
|Google only hits robots and index
Google only hits robots and index
|10||rostom||3:51 pm Jul 15, 2005|
|Disallowing & in url
||2||Wolvereness||2:54 pm Jul 12, 2005|
|robots.txt verses sitemap.html verses OPML file
||2||Neal||8:39 pm Jul 11, 2005|
|Just added robots.txt file
||4||soccrt||4:41 am Jul 11, 2005|
|block all except
||2||Radial||9:31 pm Jul 7, 2005|
|robots.txt with? in URL
||16||jozomannen||11:17 am Jul 6, 2005|
|How do i add robot.txt?
||10||genny||4:38 am Jul 6, 2005|
|Include files, spiders, and robots.txt
||4||rottsnroses||11:24 pm Jul 3, 2005|
|How to Stop BecomeBot?
||12||shilmy||3:09 pm Jun 29, 2005|
|Google ignoring my robots.txt?
||9||homesby||12:17 am Jun 29, 2005|
|Robots meta tag *CORRECT* syntax
Space* or no space for "index,*follow", and case sensitive?
|6||Clint||12:56 pm Jun 28, 2005|
|Question about Robots with multiple similar sites
Question about Robots with multiple similar sites
|2||ccash||9:54 pm Jun 27, 2005|
|Robots/Slurp, etc - what does this mean
I am a newbie - not sure what this means
|4||ramnyc||3:01 pm Jun 22, 2005|
|Does adding robot text help your site get crawled more often?
Advantages of robot text?
|6||moneyangel13||8:04 am Jun 22, 2005|
|wildcards in page names
|5||jimsthoughts||3:54 pm Jun 21, 2005|
|How can I allow robots to access my pages
inktomi is following links it should not
|4||expat123||10:08 am Jun 21, 2005|
|Updating robots.txt and Crawlers behaviour
How to tell bots to come back when you changed robots.txt banning them
|6||silverbytes||6:57 pm Jun 20, 2005|
|slurp ignoring disallow?
-or have i got it wrong?
|4||tallis||12:09 pm Jun 20, 2005|
|Question about blocking 1 robot, but allow all others
||8||capsmaster||3:26 pm Jun 18, 2005|