Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|robots.txt verses sitemap.html verses OPML file
||2||Neal||8:39 pm Jul 11, 2005|
|Just added robots.txt file
||4||soccrt||4:41 am Jul 11, 2005|
|block all except
||2||Radial||9:31 pm Jul 7, 2005|
|robots.txt with? in URL
||16||jozomannen||11:17 am Jul 6, 2005|
|How do i add robot.txt?
||10||genny||4:38 am Jul 6, 2005|
|Include files, spiders, and robots.txt
||4||rottsnroses||11:24 pm Jul 3, 2005|
|How to Stop BecomeBot?
||12||shilmy||3:09 pm Jun 29, 2005|
|Google ignoring my robots.txt?
||9||homesby||12:17 am Jun 29, 2005|
|Robots meta tag *CORRECT* syntax
Space* or no space for "index,*follow", and case sensitive?
|6||Clint||12:56 pm Jun 28, 2005|
|Question about Robots with multiple similar sites
Question about Robots with multiple similar sites
|2||ccash||9:54 pm Jun 27, 2005|
|Robots/Slurp, etc - what does this mean
I am a newbie - not sure what this means
|4||ramnyc||3:01 pm Jun 22, 2005|
|Does adding robot text help your site get crawled more often?
Advantages of robot text?
|6||moneyangel13||8:04 am Jun 22, 2005|
|wildcards in page names
|5||jimsthoughts||3:54 pm Jun 21, 2005|
|How can I allow robots to access my pages
inktomi is following links it should not
|4||expat123||10:08 am Jun 21, 2005|
|Updating robots.txt and Crawlers behaviour
How to tell bots to come back when you changed robots.txt banning them
|6||silverbytes||6:57 pm Jun 20, 2005|
|slurp ignoring disallow?
-or have i got it wrong?
|4||tallis||12:09 pm Jun 20, 2005|
|Question about blocking 1 robot, but allow all others
||8||capsmaster||3:26 pm Jun 18, 2005|
|Frequencey of GET /robots.txt
How often should a crawler/spider/robot ask for robots.txt?
|9||Dijkgraaf||5:08 am Jun 18, 2005|
|Need Clarification On Directory Exclusions
||10||jonrichd||5:00 am Jun 18, 2005|
|Robots.txt and Mod Rewrite
Which url does the robot read?
|2||eflouret||4:52 am Jun 18, 2005|
|how to ban Google from indexing ALL .php files
I'm using rewrite and don't want a dupe penalty
|16||walkman||5:12 pm Jun 15, 2005|
|boitho.com bot violating robots.txt
Specifically requested only forbidden files ( 1 2 3 4 )
|111||jazzguy||8:30 pm Jun 14, 2005|
|robots.txt and Google on intranet
robots.txt not working
|4||John488||8:11 pm Jun 13, 2005|
|Specific Spiders do need more bandwidth?
Leaving Gbot or Msn does other robots take more bandwidth when crawling?
|2||bnvrc||12:39 pm Jun 13, 2005|
|robots.text ..please help
help with robots.text file
|2||dmiller63||12:35 pm Jun 13, 2005|