Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Importance of Hotbot |
5 | slobizman | 9:08 am Nov 20, 2003 | ||||
| How long until robots.txt works? |
11 | slobizman | 3:23 pm Nov 18, 2003 | ||||
| Unknown robots crawl, spider, and robot. |
2 | synergy | 4:10 am Nov 16, 2003 | ||||
| Would crawlers crawl this? Usiing the correct robots.txt technique |
4 | ulysee | 3:02 pm Nov 15, 2003 | ||||
| internetseer billing for requested data |
14 | plumsauce | 7:47 pm Nov 14, 2003 | ||||
| Pros and cons of having a robots.txt file? |
3 | ideavirus | 5:32 pm Nov 13, 2003 | ||||
| Does anyone know what this is? I am thinking virus infected machine but unsure |
3 | Crowbar | 4:28 pm Nov 12, 2003 | ||||
| Blocking .log file with robots.txt Is it necessary, and syntax...? |
7 | Robert_Charlton | 6:59 am Nov 11, 2003 | ||||
| How to make a page *inaccessible* to searches without using password protection? |
12 | halfandhalf | 8:04 pm Nov 10, 2003 | ||||
| the wrong way to do robots.txt kinda funny, whitehouse.gov robots file |
13 | amznVibe | 7:42 pm Nov 10, 2003 | ||||
| Why bother Is it worth blocking spambots with robots.txt |
4 | WyrdoManx | 11:52 pm Nov 8, 2003 | ||||
| Does content type matter? is text/html okay |
2 | kevinsl | 8:27 am Nov 8, 2003 | ||||
| IE "Synchronize Offline while idle" is killing me Synchronize Offline |
3 | Helice | 12:46 am Nov 6, 2003 | ||||
| Bots ignoring my robots.txt In place for a week, validates, but no use |
12 | roddy | 7:10 am Nov 5, 2003 | ||||
| Blocking a spider from a directory |
3 | JVB_Mktg | 4:30 am Nov 5, 2003 | ||||
| Robots.txt To use... or not to use? That is the question |
6 | panic | 1:29 am Nov 5, 2003 | ||||
| Robots.txt Validator We're sorry, this robots.txt does NOT validate. |
2 | midge | 12:48 pm Nov 3, 2003 | ||||
| Robot.txt - What is it, and why use it |
4 | laffe | 4:36 am Nov 2, 2003 | ||||
| robots.txt syntex How to block a directory. |
6 | jdancing | 7:52 pm Oct 31, 2003 | ||||
| Robots.txt during development Anyone advise on this? |
2 | stuntdubl | 3:30 am Oct 30, 2003 | ||||
| bad Bots |
3 | cgj5204 | 4:40 am Oct 29, 2003 | ||||
| Restrict Specific Domain Names from Being Indexed Using the robots.txt file on Google to restrict certain domains... |
3 | jmh1970 | 1:36 am Oct 29, 2003 | ||||
| New Forum : Robots.txt |
1 | Brett_Tabke | 9:24 pm Oct 28, 2003 | ||||
| Put your robots.txt on a diet Three easy ways to reduce bloat in robots.txt |
20 | jdMorgan | 8:36 pm Oct 28, 2003 | ||||
| Blocking image file indexing without robots.txt Can this be done? |
2 | Winedark | 8:08 pm Oct 28, 2003 |
