Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Robots.txt and frontpage
||14||sunflower12||1:32 pm Sep 4, 2005|
|The codes required to call the robots
||2||benifer||10:04 pm Aug 31, 2005|
|Google Hammering My site (1500+ hits already today)
How to edit robots.txt to set delay
|5||wfernley||12:05 pm Aug 31, 2005|
|Why do you need robots.txt?
||5||gordele||6:38 pm Aug 30, 2005|
|Specifying pages for robots to crawl
Worth it or not?
|1||sixdeep||6:14 pm Aug 29, 2005|
|Including specific robots necessary or not?
||7||silverbytes||8:27 am Aug 29, 2005|
|how do i ban image robots
||2||spivei||11:04 pm Aug 28, 2005|
|basic idea of robots.txt
||8||moi_britphile||10:38 am Aug 28, 2005|
|Banning AskJeeves Bot
Need to ban askJeeves
|2||Marcus_Aurelius||8:31 pm Aug 27, 2005|
|problem with robots.txt file
problem with robots.txt file
|5||raybo75||5:36 pm Aug 27, 2005|
|How Often Should a Spider Come
What is the right amount of access a spider should have
|4||sirjeffer||11:00 pm Aug 22, 2005|
|Bot cn exclude this directory?
||2||tintin74||10:42 pm Aug 22, 2005|
|disallow all : including host
||4||moua||3:57 pm Aug 22, 2005|
|Google Sitemap file name?
What should I call my Google Sitemap?
|2||jetnovo||3:21 pm Aug 22, 2005|
|Spiders and redirects to disallowed directories
Are the disallowed pages indexed?
|2||cigjonser||7:18 pm Aug 19, 2005|
|allow some and exclude the rest in robots.txt
||9||jcmiras||5:24 am Aug 19, 2005|
|can you have 2 robots.txt files on site?
one in main and one in subdomain
|8||stiksandstones||12:10 am Aug 19, 2005|
|What bots ignore robots.txt?
||8||Jon12345||10:56 pm Aug 18, 2005|
Need Help Deciphering Validator Output
|7||Jioffri||6:07 am Aug 18, 2005|
|Quick question on robots.txt
Can you separate the agent and the allows/disallows?
|2||cmendla||11:19 pm Aug 17, 2005|
|Google's Nonstandard robots.txt is Not Valid
Googles robots.txt Does not validate
|6||forak||1:59 pm Aug 17, 2005|
|Search engines picking up our backend?
Developement site picked up
|3||financialhost||12:46 pm Aug 11, 2005|
|Still no PR for site
||3||gnuman||8:39 pm Aug 10, 2005|
|Can excluding with one word also exclude phrases?
||3||Marcia||10:47 pm Aug 7, 2005|
|New algo about Robots.txt
Is there any new algo introduced regarding robots.txt?
|4||Graham_Hans||2:31 am Aug 7, 2005|