Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| robots.txt -- block one user-agent, allow all others |
2 | luma | 1:44 am May 23, 2002 | ||||
| Robot.txt file help How to create this file |
7 | beasscr | 6:30 pm May 16, 2002 | ||||
| robots.txt Generates a 404 error in my logs |
9 | zeus | 1:52 pm May 3, 2002 | ||||
| Does robots.txt override meta tag? |
8 | kira | 7:12 pm May 2, 2002 | ||||
| Robots.txt validator updated. |
7 | Brett_Tabke | 6:48 pm Apr 19, 2002 | ||||
| What is robots.txt syntax |
22 | Alec_Doggone | 9:07 am Apr 19, 2002 | ||||
| Submitting robots.txt to google Would google look sooner for it? |
17 | Yidaki | 8:19 am Apr 19, 2002 | ||||
| Robots.txt Pattern Matching How Well Does It Work? |
5 | WebGuerrilla | 1:15 pm Mar 15, 2002 | ||||
| Robots.txt Question? How often does Ink check for the file?? |
6 | MarkHutch | 5:11 pm Mar 11, 2002 | ||||
| Do I need a robots.txt file in my directory? Google Bot - Whats Up with it? |
10 | jady | 9:58 pm Feb 24, 2002 | ||||
| How important is the Robots.txt file now? ....in 2002 |
29 | ScottM | 1:16 am Jan 28, 2002 | ||||
| Robots.txt Question |
7 | mud | 3:21 am Jan 17, 2002 | ||||
| I am writing my 1st robots.txt :) - never bothered before Anyone know any crawlers I should add ?? |
3 | diddlydazz | 4:11 am Jan 12, 2002 | ||||
| MSIECrawler not obeying my robots.txt :( any ideas ? |
5 | diddlydazz | 3:09 am Jan 12, 2002 | ||||
| Robots.txt standard |
4 | count_zer0 | 11:21 am Dec 7, 2001 | ||||
| robots.txt |
7 | Kenny B | 1:02 pm Sep 25, 2001 | ||||
| Robots.txt specific query |
8 | IanTurner | 7:19 pm Sep 24, 2001 | ||||
| basic robots.txt question |
5 | kapow | 4:31 pm Aug 2, 2001 | ||||
| I am new to all of this about Spiders...Robot.txt Help Message |
8 | lightwriter | 12:42 am Jul 23, 2001 | ||||
| robots.txt Banning dodgy spiders - do they take any notice? |
4 | themoff | 7:35 pm Jul 13, 2001 | ||||
| Robots.txt Does Google honour it.... |
12 | Rusky | 5:58 am Jul 5, 2001 | ||||
| Fear and Loathing of Robots.txt Coming Clean - Confessions of a a robo'phobic |
12 | Brett_Tabke | 1:23 am Jun 25, 2001 | ||||
| Bad style in robots.txt file. Why? |
5 | billyblu | 10:25 am Jun 1, 2001 | ||||
| Robots.txt Validator |
4 | Froggyman | 9:17 am May 26, 2001 | ||||
| Robots.txt Is there such a thing as Allow: * |
4 | merlin78 | 2:38 am May 24, 2001 |
