Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| New to robots.txt and how spider find robots |
9 | jazzvn | 11:49 pm Apr 10, 2004 | ||||
| Will the following robots.txt disallow all robots BUT Google? |
3 | georgec | 1:15 pm Apr 9, 2004 | ||||
| Bot blocking best way to accomplish |
2 | IONWeb | 7:18 am Apr 8, 2004 | ||||
| new to robots is this right? |
6 | sara2 | 5:38 pm Apr 5, 2004 | ||||
| Hiding robot.txt How to prevent robot.txt from beeing read by visitors |
7 | Spazio1999 | 1:00 pm Apr 3, 2004 | ||||
| Googlebot GET /robots.txt HTTP/1.0" 404 Googlebot is the only bot getting 404 |
1 | eesnard | 10:52 pm Apr 2, 2004 | ||||
| mozDex is spidering FYI on the user agent |
2 | ByronM | 9:18 pm Apr 1, 2004 | ||||
| WebmasterWorld disallowing Mozilla, Why? If you look at the WebmasWorld Robots.txt, Mozilla is disallowed, why is that |
5 | bufferzone | 10:59 pm Mar 31, 2004 | ||||
| What does "root of domain" mean & what is 404 error? root of domain 404 error |
2 | motherhen | 8:14 am Mar 31, 2004 | ||||
| Will this robots.txt do it? |
3 | rover | 5:10 am Mar 30, 2004 | ||||
| disallow: /* robot.txt newbie, please help |
2 | emm0uk | 11:15 pm Mar 29, 2004 | ||||
| robots.txt blocking foreign search engines google robots.txt block foreign engines |
1 | jer61375 | 4:49 pm Mar 29, 2004 | ||||
| Allowing only search engines to crawl forums Only want search engines in my forums |
7 | Psycho111 | 8:40 pm Mar 27, 2004 | ||||
| does a search engine care if a robots.txt file is 403 instead of 404 |
4 | kahuna | 6:20 pm Mar 27, 2004 | ||||
| robots.txt and multiple domain names Using a Virtual Server... |
1 | nerolabs | 7:45 pm Mar 26, 2004 | ||||
| google images ignoring disallow |
2 | apollo | 10:43 pm Mar 25, 2004 | ||||
| Validation Error Ran the robots.txt validation checker and recieved nothing but errors. |
3 | jseamless | 5:06 pm Mar 25, 2004 | ||||
| Why use Robots - Again? What are the benefits of using Robots. |
4 | oltc | 4:48 pm Mar 25, 2004 | ||||
| How can I block Cyveillancebot(63.148.99.****) It doesn't look like a useful bot... |
6 | itisgene | 4:43 pm Mar 25, 2004 | ||||
| Yahoo-MMCrawler |
1 | simondev3 | 8:08 pm Mar 24, 2004 | ||||
| Referral log spamming: How use robots.txt to keep them out? |
10 | Dan_Vendel | 12:36 pm Mar 24, 2004 | ||||
| robots.txt elementary question |
3 | Vespasian | 5:29 am Mar 24, 2004 | ||||
| How to identify search engine robots? What's the best way for ASP developers to identify robots? |
1 | vieth | 3:48 pm Mar 23, 2004 | ||||
| Robot text for dissallow asp pages indexed |
2 | DivinoStore | 8:06 am Mar 21, 2004 | ||||
| Anyone recognize this agent? Can't find any info about it anywhere |
2 | martyt | 4:15 pm Mar 20, 2004 |
