Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Do i need the txt file? if i want to allow everyone |
4 | markbiz | 12:59 am July 5, 2004 | ||||
| I'm not understanding the different makes of bots Can I stop some, while allowing others? |
2 | tomequin | 10:11 pm July 1, 2004 | ||||
| Yahoo! Slurp - robots.txt Not obeyng robots.txt |
5 | EarWig | 1:59 pm June 29, 2004 | ||||
| Making robots? |
4 | lifedeuce | 1:22 pm June 29, 2004 | ||||
| Disallow pages in a subdirectory Basic help please |
3 | pjuk | 8:40 pm June 24, 2004 | ||||
| site setup for spider? |
5 | Gemelli_no1 | 12:51 pm June 23, 2004 | ||||
| needed? is it completely neccisary |
3 | natty | 1:34 pm June 15, 2004 | ||||
| Robots dot text for all user files? robots.txt all public_html |
4 | kj6loh | 7:21 pm June 11, 2004 | ||||
| wildcards in the Disallow field, non standard? |
3 | pmac | 4:19 pm June 11, 2004 | ||||
| where to place robots.txt? robots newbie alert |
6 | Duke_of_Url | 10:17 pm June 10, 2004 | ||||
| How should I use a robots.txt file on a virtual account? I want to ban spiders from one site only |
3 | soccer_star | 2:01 pm June 10, 2004 | ||||
| Robots.txt for yahoo and google.com google and yahoo robots.txt |
3 | adisakmai | 4:47 am June 10, 2004 | ||||
| Prevent Google's "cached" link from appearing but still stay in the search results |
4 | digitalv | 4:51 pm June 8, 2004 | ||||
| Disallowing specific query strings |
3 | duke33 | 3:05 pm June 8, 2004 | ||||
| Is it a spider? |
2 | topher | 5:38 am June 8, 2004 | ||||
| Preventing robot access to all subdirectories within a directory |
2 | Leonardi | 12:16 am June 7, 2004 | ||||
| Robots.txt triggers 404 error Robots.txt triggers 404 error |
2 | Nout | 11:10 am May 29, 2004 | ||||
| Is this robots.txt ok? |
5 | esllou | 12:32 pm May 26, 2004 | ||||
| Removing "query-stringed" pages from Google And other search engines using robotx.txt |
3 | MatthewHSE | 2:23 pm May 25, 2004 | ||||
| Stop Spiders Crawling Message Board Edit Cgi Factory Message Board |
7 | mn1dbp | 12:36 pm May 25, 2004 | ||||
| Google still lists disallowed pages How long do disallowed pages stay in the index? |
5 | jam13 | 10:56 am May 23, 2004 | ||||
| large robots.txt file What size will cause a crawler to choke in the file |
7 | DoppyNL | 9:43 pm May 21, 2004 | ||||
| Using robot.txt to keep unwanted backlinking Is this an effective means |
2 | Daily_Sparring | 5:14 pm May 21, 2004 | ||||
| importance of a robots.txt file |
11 | hughie | 12:27 am May 21, 2004 | ||||
| Logic Error Wildcard userAgent last entry in robots.txt file? |
2 | pageoneresults | 2:34 pm May 20, 2004 |
