Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Newbie robots.txt question
Can a robot find content not referenced in pages on the site?
|4||joykesilon||8:02 pm Mar 6, 2006|
|Disallowing double bot crawl due to addon domain
||2||Alex_lon||8:04 pm Mar 2, 2006|
|Robots.txt Basic question
|3||omoutop||9:46 am Feb 27, 2006|
|3||malasorte||6:10 pm Feb 26, 2006|
How do I get the bots to search my content instead of my intro page?
|3||xmar||4:32 am Feb 25, 2006|
|Blocking Content On A New Site
New version of major website being launched.
|2||caragh||3:04 pm Feb 23, 2006|
|htaccess Allow robots.txt and leech protection
||3||war59312||3:30 am Feb 23, 2006|
|NimbleCrawler is Bad Bot, well worth banning.
NimbleCrawler is Bad Bot. Never reads robots.txt file.
|10||d_mot||1:24 am Feb 23, 2006|
|In the era of increasing numbers of bad bots is robots.txt irrelevant?
Why bother? Why not go the other way and white-list with controls? ( 1 2 )
|34||Webwork||9:21 pm Feb 21, 2006|
|Disallow a directory, but allow one of its subdirectories
||2||Nutter||9:48 am Feb 21, 2006|
How do I ensure re-direct pages are not duplicated?
|1||robinantill||11:05 pm Feb 18, 2006|
|Complicated Robots.txt / htaccess Question
|1||fraggle||9:23 pm Feb 18, 2006|
|robots.txt - subdomains and domain
where to place
|3||Rose_1171||8:06 am Feb 18, 2006|
|Http: vs Https:
What are the best ways to stop crawlers?
|3||thaedge||7:54 am Feb 18, 2006|
|Does robots.txt work per directory or per domain?
||3||zver0||6:40 pm Feb 16, 2006|
|Do bots confuse .htm and .html files in robots.txt?
||2||subgenius||6:08 pm Feb 16, 2006|
|robots.txt Advertising Methods
The new trend...
|3||pageoneresults||4:49 pm Feb 16, 2006|
|Banning all subdirectories in a directory without the files?
||5||zver0||1:28 pm Feb 16, 2006|
|renewed site not being indexed - old pages in serp cache
how to fix robots.txt mistake
|4||Ring||10:47 am Feb 15, 2006|
|Why doesn't this validate?
|4||Vimes||11:04 am Feb 14, 2006|
||1||jexx||9:02 pm Feb 13, 2006|
|how to disallow shopping cart
rank is dropping since we set it up
|2||Lorel||7:39 am Feb 12, 2006|
Ok, maybe I did not understand how it works
|15||followgreg||9:20 pm Feb 10, 2006|
|? and * in robots.txt
What stands a questionmark in the robots.txt for?
|2||solandre||5:50 pm Feb 9, 2006|
|Meta tags and Robots
||5||Buster42||4:48 pm Feb 9, 2006|