Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Banning AskJeeves Bot
Need to ban askJeeves
|2||Marcus_Aurelius||8:31 pm Aug 27, 2005|
|problem with robots.txt file
problem with robots.txt file
|5||raybo75||5:36 pm Aug 27, 2005|
|How Often Should a Spider Come
What is the right amount of access a spider should have
|4||sirjeffer||11:00 pm Aug 22, 2005|
|Bot cn exclude this directory?
||2||tintin74||10:42 pm Aug 22, 2005|
|disallow all : including host
||4||moua||3:57 pm Aug 22, 2005|
|Google Sitemap file name?
What should I call my Google Sitemap?
|2||jetnovo||3:21 pm Aug 22, 2005|
|Spiders and redirects to disallowed directories
Are the disallowed pages indexed?
|2||cigjonser||7:18 pm Aug 19, 2005|
|allow some and exclude the rest in robots.txt
||9||jcmiras||5:24 am Aug 19, 2005|
|can you have 2 robots.txt files on site?
one in main and one in subdomain
|8||stiksandstones||12:10 am Aug 19, 2005|
|What bots ignore robots.txt?
||8||Jon12345||10:56 pm Aug 18, 2005|
Need Help Deciphering Validator Output
|7||Jioffri||6:07 am Aug 18, 2005|
|Quick question on robots.txt
Can you separate the agent and the allows/disallows?
|2||cmendla||11:19 pm Aug 17, 2005|
|Google's Nonstandard robots.txt is Not Valid
Googles robots.txt Does not validate
|6||forak||1:59 pm Aug 17, 2005|
|Search engines picking up our backend?
Developement site picked up
|3||financialhost||12:46 pm Aug 11, 2005|
|Still no PR for site
||3||gnuman||8:39 pm Aug 10, 2005|
|Can excluding with one word also exclude phrases?
||3||Marcia||10:47 pm Aug 7, 2005|
|New algo about Robots.txt
Is there any new algo introduced regarding robots.txt?
|4||Graham_Hans||2:31 am Aug 7, 2005|
Is the dmoz.org/world disallowed?
|4||angiolo||8:05 am Aug 6, 2005|
|How to exclude Oracle Portal Help pages using robots.txt
Need to exclude Oracle Portal Help Pages from search engines
|10||atul_iiit||9:12 am Aug 5, 2005|
|http://<hostname.domainname>/robots.txt is access denied can robots
Can robots read robots.txt if I get access denied for this file
|4||atul_iiit||4:25 pm Aug 4, 2005|
What's the importance of Robot.txt?
|3||Lowkei||9:02 am Jul 31, 2005|
|Don't want to index dynamic pages
But would it be a mistake to block them?
|5||chopin2256||2:12 am Jul 30, 2005|
|Could 744 permissions on robots.txt cause problems
all other files are set to 644
|5||dbar||7:41 pm Jul 29, 2005|
viewing the robots.txt file
|2||eleap||3:27 pm Jul 27, 2005|
|How to exclude a few pages?
||2||jack38||10:25 am Jul 27, 2005|