Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
Anyone know of one?
|7||thaedge||11:01 pm Apr 5, 2006|
|how to disallow spider to index one of my subdomain' files?
||4||peony||7:44 pm Apr 5, 2006|
|Sessions & robots - need help
||3||zuses||6:54 am Mar 31, 2006|
what is libwww-perl/5.803
|4||evo81||8:12 pm Mar 28, 2006|
|robots -how to disallow search results
||5||zuses||10:53 am Mar 23, 2006|
|Syntax check robots.txt
There should be atleast 1 disallow line in any Robots.txt.
|3||mycutegoddess||10:32 am Mar 17, 2006|
|robots.txt in PHP Nuke
I need help with create robots.txt in PHP Nuke
|2||spediter||2:28 am Mar 16, 2006|
|What is robots.txt?
|4||forex||6:01 am Mar 14, 2006|
Research on valid robots.txt and usage
|4||smitha_ajay||5:18 am Mar 14, 2006|
disallow external link
|7||dennis_oppelt||2:35 am Mar 14, 2006|
|more on robots and mirror sites
|3||dennis_oppelt||12:48 am Mar 14, 2006|
||3||OldWolf||5:40 pm Mar 13, 2006|
|How do I use a wild card for many pages?
|6||maha||1:38 am Mar 12, 2006|
|Elimination of pages from a website.
Do i have to notify the engines in some way?
|9||Rani||12:26 am Mar 12, 2006|
|Robots.txt and htm - html
is disallow for htm killing pages that are html?
|5||chewy||6:37 pm Mar 8, 2006|
|Newbie robots.txt question
Can a robot find content not referenced in pages on the site?
|4||joykesilon||8:02 pm Mar 6, 2006|
|Disallowing double bot crawl due to addon domain
||2||Alex_lon||8:04 pm Mar 2, 2006|
|Robots.txt Basic question
|3||omoutop||9:46 am Feb 27, 2006|
|3||malasorte||6:10 pm Feb 26, 2006|
How do I get the bots to search my content instead of my intro page?
|3||xmar||4:32 am Feb 25, 2006|
|Blocking Content On A New Site
New version of major website being launched.
|2||caragh||3:04 pm Feb 23, 2006|
|htaccess Allow robots.txt and leech protection
||3||war59312||3:30 am Feb 23, 2006|
|NimbleCrawler is Bad Bot, well worth banning.
NimbleCrawler is Bad Bot. Never reads robots.txt file.
|10||d_mot||1:24 am Feb 23, 2006|
|In the era of increasing numbers of bad bots is robots.txt irrelevant?
Why bother? Why not go the other way and white-list with controls? ( 1 2 )
|34||Webwork||9:21 pm Feb 21, 2006|
|Disallow a directory, but allow one of its subdirectories
||2||Nutter||9:48 am Feb 21, 2006|