Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
||6||ag_47||4:58 am Jan 8, 2009|
|Indexing and Crawling to Dynamic pages
Would google Crawl Dynamic CMS
|3||mick_simon||2:55 am Jan 7, 2009|
|Disallow in Robots against guidelines?
||4||cgrant||4:28 pm Jan 6, 2009|
|How to block this type of URL's
||9||iCyborg||9:32 pm Jan 4, 2009|
|Having Trouble With an Allow Instruction.
Trying to adapt an example given by jdMorgan.
|11||Durnovaria||1:43 pm Jan 2, 2009|
|If a dir is disallowed, will a sitemap in a subdir be read?
Sitemaps located in non-content directories
|7||badbadmonkey||8:45 pm Jan 1, 2009|
||13||FiRe||11:34 am Dec 22, 2008|
|I want entire site indexed
||7||brnm98105||8:38 pm Dec 19, 2008|
||2||fishingfriend||5:45 pm Dec 15, 2008|
|How to block a subdomian
Where to put the file
|5||Musicarl||5:39 am Dec 15, 2008|
|A bit of help if possible would be appreciated.
||8||glennk||7:52 pm Dec 14, 2008|
|Blocking Email and Print only URLS
||2||himalayaswater||12:14 am Dec 7, 2008|
|Robots.txt - Site in a Directory
||10||Sarao||10:14 pm Dec 6, 2008|
|nofollow noindex or disallow?
||4||Gemini23||12:09 am Dec 6, 2008|
|Allowing Adsence bot
Googled emailed me asking to allow the adsence bot
|5||alahamdan||2:02 pm Dec 5, 2008|
|Possible to block all urls with ? in them?
||2||texasville||1:28 pm Dec 5, 2008|
|How is correct code within robots.txt?
||4||toplisek||11:07 am Nov 29, 2008|
|disallowing a variable
||4||KevinC||10:37 pm Nov 17, 2008|
|Queries about Robots.txt
Importance of Robots.txt
|2||Monalisa||5:24 pm Nov 17, 2008|
|Disallow Yahoo to block site explorer
||9||minnapple||2:23 am Nov 14, 2008|
|Matching patterns in robots.txt
||3||kiransarv||2:39 pm Nov 9, 2008|
|how to block ALL bots?
||18||Makaveli2007||10:50 am Nov 9, 2008|
|Google has a new robots.txt for today only!
||4||travelin_cat||8:56 pm Oct 31, 2008|
|Preventing a page from being indexed but wanting links to be followed
||3||realmaverick||5:24 pm Oct 21, 2008|
|Disallow .php in root folder but allow in sub.
||3||mysticalsock||5:19 pm Oct 21, 2008|