Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Robots.txt and frontpage |
14 | sunflower12 | 1:32 pm Sept 4, 2005 | ||||
| The codes required to call the robots |
2 | benifer | 10:04 pm Aug 31, 2005 | ||||
| Google Hammering My site (1500+ hits already today) How to edit robots.txt to set delay |
5 | wfernley | 12:05 pm Aug 31, 2005 | ||||
| Why do you need robots.txt? |
5 | gordele | 6:38 pm Aug 30, 2005 | ||||
| Specifying pages for robots to crawl Worth it or not? |
1 | sixdeep | 6:14 pm Aug 29, 2005 | ||||
| Including specific robots necessary or not? |
7 | silverbytes | 8:27 am Aug 29, 2005 | ||||
| how do i ban image robots |
2 | spivei | 11:04 pm Aug 28, 2005 | ||||
| basic idea of robots.txt |
8 | moi_britphile | 10:38 am Aug 28, 2005 | ||||
| Banning AskJeeves Bot Need to ban askJeeves |
2 | Marcus_Aurelius | 8:31 pm Aug 27, 2005 | ||||
| problem with robots.txt file problem with robots.txt file |
5 | raybo75 | 5:36 pm Aug 27, 2005 | ||||
| How Often Should a Spider Come What is the right amount of access a spider should have |
4 | sirjeffer | 11:00 pm Aug 22, 2005 | ||||
| Bot cn exclude this directory? |
2 | tintin74 | 10:42 pm Aug 22, 2005 | ||||
| disallow all : including host |
4 | moua | 3:57 pm Aug 22, 2005 | ||||
| Google Sitemap file name? What should I call my Google Sitemap? |
2 | jetnovo | 3:21 pm Aug 22, 2005 | ||||
| Spiders and redirects to disallowed directories Are the disallowed pages indexed? |
2 | cigjonser | 7:18 pm Aug 19, 2005 | ||||
| allow some and exclude the rest in robots.txt |
9 | jcmiras | 5:24 am Aug 19, 2005 | ||||
| can you have 2 robots.txt files on site? one in main and one in subdomain |
8 | stiksandstones | 12:10 am Aug 19, 2005 | ||||
| What bots ignore robots.txt? |
8 | Jon12345 | 10:56 pm Aug 18, 2005 | ||||
| Validator Questions Need Help Deciphering Validator Output |
7 | Jioffri | 6:07 am Aug 18, 2005 | ||||
| Quick question on robots.txt Can you separate the agent and the allows/disallows? |
2 | cmendla | 11:19 pm Aug 17, 2005 | ||||
| Google's Nonstandard robots.txt is Not Valid Googles robots.txt Does not validate |
6 | forak | 1:59 pm Aug 17, 2005 | ||||
| Search engines picking up our backend? Developement site picked up |
3 | financialhost | 12:46 pm Aug 11, 2005 | ||||
| Still no PR for site |
3 | gnuman | 8:39 pm Aug 10, 2005 | ||||
| Can excluding with one word also exclude phrases? |
3 | Marcia | 10:47 pm Aug 7, 2005 | ||||
| New algo about Robots.txt Is there any new algo introduced regarding robots.txt? |
4 | Graham_Hans | 2:31 am Aug 7, 2005 |
