Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Newbie robots.txt question Can a robot find content not referenced in pages on the site? |
4 | joykesilon | 8:02 pm Mar 6, 2006 | ||||
| Disallowing double bot crawl due to addon domain |
2 | Alex_lon | 8:04 pm Mar 2, 2006 | ||||
| Robots.txt Basic question 600 rules |
3 | omoutop | 9:46 am Feb 27, 2006 | ||||
| robots.txt Help disallow issue |
3 | malasorte | 6:10 pm Feb 26, 2006 | ||||
| PHPBB Site How do I get the bots to search my content instead of my intro page? |
3 | xmar | 4:32 am Feb 25, 2006 | ||||
| Blocking Content On A New Site New version of major website being launched. |
2 | caragh | 3:04 pm Feb 23, 2006 | ||||
| htaccess Allow robots.txt and leech protection |
3 | war59312 | 3:30 am Feb 23, 2006 | ||||
| NimbleCrawler is Bad Bot, well worth banning. NimbleCrawler is Bad Bot. Never reads robots.txt file. |
10 | d_mot | 1:24 am Feb 23, 2006 | ||||
| In the era of increasing numbers of bad bots is robots.txt irrelevant? Why bother? Why not go the other way and white-list with controls?[2] ( 1 2 ) |
34 | Webwork | 9:21 pm Feb 21, 2006 | ||||
| Disallow a directory, but allow one of its subdirectories |
2 | Nutter | 9:48 am Feb 21, 2006 | ||||
| robots.txt disallow How do I ensure re-direct pages are not duplicated? |
1 | robinantill | 11:05 pm Feb 18, 2006 | ||||
| Complicated Robots.txt / htaccess Question maybe |
1 | fraggle | 9:23 pm Feb 18, 2006 | ||||
| robots.txt - subdomains and domain where to place |
3 | Rose_1171 | 8:06 am Feb 18, 2006 | ||||
| Http: vs Https: What are the best ways to stop crawlers? |
3 | thaedge | 7:54 am Feb 18, 2006 | ||||
| Does robots.txt work per directory or per domain? |
3 | zver0 | 6:40 pm Feb 16, 2006 | ||||
| Do bots confuse .htm and .html files in robots.txt? |
2 | subgenius | 6:08 pm Feb 16, 2006 | ||||
| robots.txt Advertising Methods The new trend... |
3 | pageoneresults | 4:49 pm Feb 16, 2006 | ||||
| Banning all subdirectories in a directory without the files? |
5 | zver0 | 1:28 pm Feb 16, 2006 | ||||
| renewed site not being indexed - old pages in serp cache how to fix robots.txt mistake |
4 | Ring | 10:47 am Feb 15, 2006 | ||||
| Why doesn't this validate? Help please |
4 | Vimes | 11:04 am Feb 14, 2006 | ||||
| TripAdvisorBot? |
1 | jexx | 9:02 pm Feb 13, 2006 | ||||
| how to disallow shopping cart rank is dropping since we set it up |
2 | Lorel | 7:39 am Feb 12, 2006 | ||||
| Robots.txt help Ok, maybe I did not understand how it works |
15 | followgreg | 9:20 pm Feb 10, 2006 | ||||
| ? and * in robots.txt What stands a questionmark in the robots.txt for? |
2 | solandre | 5:50 pm Feb 9, 2006 | ||||
| Meta tags and Robots |
5 | Buster42 | 4:48 pm Feb 9, 2006 |
