Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Only those on the list are allowed entrance All others OUT |
7 | grnidone | 5:29 pm Jan 2, 2004 | ||||
| txt text for website txt |
4 | krikri | 5:11 pm Jan 2, 2004 | ||||
| Check my robots.txt please Don't want to block adsense bot |
3 | conroy | 2:10 pm Jan 2, 2004 | ||||
| How to ban YahooSeeker? Used robots file, but it doesn't obey. |
2 | zf007 | 3:59 am Jan 1, 2004 | ||||
| deny a spider from visiting a single page |
3 | Michael_Weir | 3:08 pm Dec 29, 2003 | ||||
| Disallow indexing on dynamic files Disallow indexing on dynamic files |
6 | premzero | 2:58 am Dec 26, 2003 | ||||
| banning "vscooter" but allowing "scooter" |
4 | keyplyr | 9:15 pm Dec 25, 2003 | ||||
| robots.txt not showing... |
4 | mubarak_zeb | 3:37 pm Dec 19, 2003 | ||||
| Getting 403 Forbidden for robots.txt Can anyone help |
11 | GodLikeLotus | 4:59 am Dec 18, 2003 | ||||
| robots.txt use and hosting provider problem robots.txt use and hosting provider problem |
2 | david_cool | 6:55 pm Dec 16, 2003 | ||||
| robots.txt on phpbb forum |
2 | RammsteinNicCage | 4:06 pm Dec 12, 2003 | ||||
| Spiders get / and /robots.txt but no more! Is my robots.txt killing me? |
6 | dwilson | 8:55 pm Dec 11, 2003 | ||||
| [404] HTTP Page robots.txt not found 404 error |
4 | Strike | 12:44 pm Dec 10, 2003 | ||||
| changing robots.txt replacing robots.txt |
2 | darex | 8:24 pm Dec 9, 2003 | ||||
| Can we prevent a form link from being spidered? form href prevent spider indexing |
3 | Frannie | 2:32 pm Dec 9, 2003 | ||||
| Robots and directory hierachies |
4 | finlay | 5:57 pm Dec 8, 2003 | ||||
| wildcard disallow for pages with varibles would this work for pages ending with?keep= : Disallow: *keep=*? |
2 | The_Rabbit | 5:25 pm Dec 8, 2003 | ||||
| disallowing based on part of a query string |
3 | jibbajabba | 4:39 pm Dec 2, 2003 | ||||
| Whois.sc Probe/Bot Prevent Snapshot Cache |
1 | CustomFit | 3:41 am Dec 2, 2003 | ||||
| Hiding robots.txt |
21 | keyplyr | 1:45 am Dec 2, 2003 | ||||
| Did you re-test? |
3 | hybridx | 2:51 pm Nov 28, 2003 | ||||
| Weird Code Return from Googlebot when robots.txt was spidered |
3 | 5x54u | 9:14 pm Nov 25, 2003 | ||||
| Problem with robots.txt |
4 | Nisse | 8:08 am Nov 25, 2003 | ||||
| robots and dynamic url How to ban robots from a dynamic url |
3 | Ledfish | 8:09 pm Nov 23, 2003 | ||||
| Googlebot Wildcard |
5 | payday | 11:26 am Nov 22, 2003 |
