Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Http: vs Https:
What are the best ways to stop crawlers?
|3||thaedge||7:54 am Feb 18, 2006|
|Does robots.txt work per directory or per domain?
||3||zver0||6:40 pm Feb 16, 2006|
|Do bots confuse .htm and .html files in robots.txt?
||2||subgenius||6:08 pm Feb 16, 2006|
|robots.txt Advertising Methods
The new trend...
|3||pageoneresults||4:49 pm Feb 16, 2006|
|Banning all subdirectories in a directory without the files?
||5||zver0||1:28 pm Feb 16, 2006|
|renewed site not being indexed - old pages in serp cache
how to fix robots.txt mistake
|4||Ring||10:47 am Feb 15, 2006|
|Why doesn't this validate?
|4||Vimes||11:04 am Feb 14, 2006|
||1||jexx||9:02 pm Feb 13, 2006|
|how to disallow shopping cart
rank is dropping since we set it up
|2||Lorel||7:39 am Feb 12, 2006|
Ok, maybe I did not understand how it works
|15||followgreg||9:20 pm Feb 10, 2006|
|? and * in robots.txt
What stands a questionmark in the robots.txt for?
|2||solandre||5:50 pm Feb 9, 2006|
|Meta tags and Robots
||5||Buster42||4:48 pm Feb 9, 2006|
Has anyone head of a character limit in robots.txt?
|7||bumpaw||8:39 am Feb 9, 2006|
|Is this Robots.txt OK?
Please check this robot.txt file
|9||mzconstruction||11:06 pm Feb 3, 2006|
|robots.txt and SSL
||3||AlpacaAnnie||8:24 am Feb 3, 2006|
|Are different User-agent entries needed for versions of similar robot
||3||Ceyx||2:29 am Feb 1, 2006|
|Robots list service?
Looking for a service provider or subscription service
|2||xbase234||1:41 am Feb 1, 2006|
|How to disallow URLs with parameters?
"/page-name.php?" or "/page-name.php"
|11||iProgram||11:34 pm Jan 31, 2006|
||3||TonysDesigns||10:13 pm Jan 30, 2006|
|block robots from images
|2||gb_dev||9:12 pm Jan 30, 2006|
|If my website dont have robots.txt
|7||rusmart||8:50 pm Jan 30, 2006|
|Duplicate Content in Robots.txt file
||2||palmpal||12:17 am Jan 30, 2006|
|To allow some robots only
||9||GoldenHammer||3:14 pm Jan 28, 2006|
|robots.txt vs htaccess for control
if the robots.txt is not dependable why not just use htaccess?
|6||ccubed99||3:39 am Jan 28, 2006|
|robots.txt and htaccess
||4||agradation||11:17 pm Jan 26, 2006|