Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Robots meta tag *CORRECT* syntax Space* or no space for "index,*follow", and case sensitive? |
6 | Clint | 12:56 pm June 28, 2005 | ||||
| Question about Robots with multiple similar sites Question about Robots with multiple similar sites |
2 | ccash | 9:54 pm June 27, 2005 | ||||
| Robots/Slurp, etc - what does this mean I am a newbie - not sure what this means |
4 | ramnyc | 3:01 pm June 22, 2005 | ||||
| Does adding robot text help your site get crawled more often? Advantages of robot text? |
6 | moneyangel13 | 8:04 am June 22, 2005 | ||||
| wildcards in page names Disallow: /store/scripts/emailFriend.asp* |
5 | jimsthoughts | 3:54 pm June 21, 2005 | ||||
| How can I allow robots to access my pages inktomi is following links it should not |
4 | expat123 | 10:08 am June 21, 2005 | ||||
| Updating robots.txt and Crawlers behaviour How to tell bots to come back when you changed robots.txt banning them |
6 | silverbytes | 6:57 pm June 20, 2005 | ||||
| slurp ignoring disallow? -or have i got it wrong? |
4 | tallis | 12:09 pm June 20, 2005 | ||||
| Question about blocking 1 robot, but allow all others |
8 | capsmaster | 3:26 pm June 18, 2005 | ||||
| Frequencey of GET /robots.txt How often should a crawler/spider/robot ask for robots.txt? |
9 | Dijkgraaf | 5:08 am June 18, 2005 | ||||
| Need Clarification On Directory Exclusions |
10 | jonrichd | 5:00 am June 18, 2005 | ||||
| Robots.txt and Mod Rewrite Which url does the robot read? |
2 | eflouret | 4:52 am June 18, 2005 | ||||
| how to ban Google from indexing ALL .php files I'm using rewrite and don't want a dupe penalty |
16 | walkman | 5:12 pm June 15, 2005 | ||||
| boitho.com bot violating robots.txt Specifically requested only forbidden files[4] ( 1 2 3 4 ) |
111 | jazzguy | 8:30 pm June 14, 2005 | ||||
| robots.txt and Google on intranet robots.txt not working |
4 | John488 | 8:11 pm June 13, 2005 | ||||
| Specific Spiders do need more bandwidth? Leaving Gbot or Msn does other robots take more bandwidth when crawling? |
2 | bnvrc | 12:39 pm June 13, 2005 | ||||
| robots.text ..please help help with robots.text file |
2 | dmiller63 | 12:35 pm June 13, 2005 | ||||
| Robot,txt Help |
8 | branmh | 6:34 pm June 12, 2005 | ||||
| Google Robots.txt Wildcard Feature Please take a look at my robot.txt |
4 | seunosewa | 7:02 am June 11, 2005 | ||||
| Separate robots.txt file for secure site How do i block one and not the other? |
3 | Vimes | 3:34 am June 10, 2005 | ||||
| Bad bots |
15 | ownerrim | 4:25 am June 8, 2005 | ||||
| If I disallow /filename.php ... will google still index /filename.php?whatever |
7 | Trisha | 7:05 am June 7, 2005 | ||||
| Agent Name for MSN Where can I find this information. |
8 | Pat1975 | 5:55 am June 3, 2005 | ||||
| General questions General robots.txt queries |
7 | makzan | 11:49 pm June 2, 2005 | ||||
| Disallow /file.php same as Diallow file.php? |
6 | One_on_One | 3:43 am June 2, 2005 |
