Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Robots.txt command? "do not search anything outside this." |
5 | OWizeOne | 1:08 pm Feb 1, 2007 | ||||
| How do I block yahoo's SE spider and allow their image spider? Does Yahoo send a different spider for the image search? |
4 | Clark | 5:18 pm Jan 27, 2007 | ||||
| How robot.txt works Robot.txt |
3 | walmslei | 4:10 am Jan 27, 2007 | ||||
| Why is google indexing this banned directory? |
5 | ichthyous | 4:55 pm Jan 26, 2007 | ||||
| https on google and robots.txt |
2 | concepts99 | 1:54 pm Jan 24, 2007 | ||||
| directory structure question |
5 | union_jack | 7:43 pm Jan 22, 2007 | ||||
| block a subdirectory |
2 | concepts99 | 2:05 pm Jan 22, 2007 | ||||
| Google not respecting robots.txt Found >500 pages in Google index that are forbiden by robots.txt |
5 | roots | 5:54 am Jan 22, 2007 | ||||
| domo arigato mr. roboto |
3 | ricnut | 11:52 pm Jan 20, 2007 | ||||
| Disallow Slurp from all files except two Yahoo likes only one page, so I'd like to just 'give' it that. |
2 | tigertom | 3:00 am Jan 20, 2007 | ||||
| Yahoo Slurp Bot misbehaving .. or is it someone masquerading? |
4 | tigertom | 9:27 am Jan 17, 2007 | ||||
| Blocking Site in some countries |
3 | eldinboy | 5:55 pm Jan 15, 2007 | ||||
| What to do with duplicate pages |
3 | olba | 2:12 am Jan 12, 2007 | ||||
| Why should I have a robots.txt file? Isn't it OK if I don't? |
23 | beren | 8:45 am Jan 11, 2007 | ||||
| Google Site Diagnostics - robots.txt blocks urls |
4 | akreider | 7:44 pm Jan 5, 2007 | ||||
| Robots txt to solve gallery duplicates can someone confirm this is correct |
3 | caines | 10:54 am Jan 5, 2007 | ||||
| How many search engine understand crawl delay? |
1 | foxfox | 3:38 pm Jan 4, 2007 | ||||
| Multiple Domains Issue |
3 | kdobson99 | 4:58 am Jan 4, 2007 | ||||
| Do spiders follow links from pages with the noindex tag? As opposed to links from robots files? |
4 | bouncybunny | 6:59 am Jan 2, 2007 | ||||
| Directing Spiders to specific Servers |
2 | TheSeoGuy | 1:50 pm Dec 29, 2006 | ||||
| I had robots.txt blocking my phpbb folders which are now deleted Should I remove that line from robots.txt now? |
2 | cmendla | 4:16 pm Dec 24, 2006 | ||||
| Disallowing Indexed files in Google |
5 | leo11877 | 2:01 pm Dec 23, 2006 | ||||
| Disallowing by date. Is it possible to keep some page and disallow new ones...? |
3 | bouncybunny | 8:31 am Dec 20, 2006 | ||||
| Worried about profile links |
4 | Asia_Expat | 3:53 pm Dec 19, 2006 | ||||
| Is it possible to exclude a URL by specifying an anchor? Excluding #...? |
3 | bouncybunny | 12:40 am Dec 17, 2006 |
