Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| need robots.txt for pages that are not linked? robots.txt |
5 | rld3 | 5:09 pm Jan 29, 2004 | ||||
| SEO Robot.txt tag and robot meta tags... need advice Help about robot.txt and robot meta tags |
3 | jamieabbott | 2:09 pm Jan 28, 2004 | ||||
| Robots.txt returning "test" Host has really messed me up |
4 | GodLikeLotus | 10:37 am Jan 28, 2004 | ||||
| Need quick help Prevent robots from clicking on PPC links |
2 | hostlead | 11:11 am Jan 27, 2004 | ||||
| how to let robot crawl the links at a page but not index this page? index, robot |
2 | johnlim | 11:39 pm Jan 26, 2004 | ||||
| Wildcard use in robots.txt Need to disallow spidering of graphics files |
2 | Probably | 4:58 am Jan 26, 2004 | ||||
| robots.txt and htaccess .htaccess circumvents to site root |
4 | stefan009 | 5:35 pm Jan 24, 2004 | ||||
| Altavista Scooter eats up too much trafic |
2 | maxidrom11 | 5:28 am Jan 24, 2004 | ||||
| Does every site need a robots.txt? I have one, but I'm not sure I need it |
2 | MatthewHSE | 4:00 pm Jan 23, 2004 | ||||
| creating robots.txt |
5 | artdog | 12:21 pm Jan 23, 2004 | ||||
| My page has never been indexed |
2 | kknusa | 6:45 am Jan 23, 2004 | ||||
| Subdomains |
2 | BBB_Guy | 6:38 am Jan 23, 2004 | ||||
| Indexed pages that are disalowed by robots.txt is this normal? |
10 | WebJoe | 7:03 pm Jan 20, 2004 | ||||
| Google ignoring robots.txt No real surprise, but a real pain |
9 | voice220 | 4:12 pm Jan 19, 2004 | ||||
| Is there any bots to block? Else than the infamous "Web Content International"? |
3 | Daemon | 11:56 pm Jan 16, 2004 | ||||
| Robots.txt help |
5 | mikemcs | 1:04 pm Jan 16, 2004 | ||||
| Who are these spiders and what do they want? |
7 | obsos | 4:51 pm Jan 14, 2004 | ||||
| Revisit instruction for robots I want to specify different revisit instruction different sections |
3 | adityamodi | 5:13 am Jan 10, 2004 | ||||
| Have I been blocking Altavista spider from spidering my site? |
6 | irock | 6:10 pm Jan 9, 2004 | ||||
| robot.txt to remove an alias Afraid of duplicated content |
2 | Ulkari | 5:48 pm Jan 8, 2004 | ||||
| Spider Names I need the names of some SEs usernames |
3 | The_Young_One | 2:47 am Jan 8, 2004 | ||||
| Blocking Includes (SSI) blocking ssi |
3 | direktor1 | 4:03 pm Jan 7, 2004 | ||||
| Subdomains How to exclude |
4 | gosman | 2:54 pm Jan 6, 2004 | ||||
| Does the leading / make any difference? Sorry, I'm new to robots.txt |
2 | AsleepATheWheel | 1:07 am Jan 6, 2004 | ||||
| faulty robots.txt? |
5 | tenaka | 2:13 pm Jan 3, 2004 |
