Forum Moderated by: goodroi
Using a robots.txt is all part of being a good SEO.
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| A new bot? Anyone else seeing this? |
4 | walrus | 6:48 am Mar 19, 2004 | ||||
| Can't prevent php files from being searched Is this possible |
1 | Jack_mcs | 1:58 pm Mar 18, 2004 | ||||
| newbie robots.txt and forum Q how to write it |
6 | buksida | 3:27 am Mar 18, 2004 | ||||
| Robot.txt File What is it's importance? |
7 | Enigmatic | 8:52 pm Mar 16, 2004 | ||||
| QuepasaCreep Agent where is it from? |
3 | barbeta | 8:17 pm Mar 16, 2004 | ||||
| Does robots exclusion prevent pr being passed |
2 | djtaverner | 5:38 pm Mar 16, 2004 | ||||
| Should I ban this user agent? |
4 | jehoshua | 5:13 pm Mar 16, 2004 | ||||
| syntax for all php exclusion |
2 | djtaverner | 4:29 pm Mar 15, 2004 | ||||
| Cannot find error Robots.txt syntax error |
5 | webapache | 5:04 pm Mar 14, 2004 | ||||
| robots.txt and mod_rewrite just double checking |
6 | hannamyluv | 2:24 am Mar 14, 2004 | ||||
| Several robots.txt? |
2 | brakkar | 1:18 am Mar 14, 2004 | ||||
| Where is the "Top Level"? is this important these days? |
4 | chewy | 9:46 pm Mar 13, 2004 | ||||
| Does this work? Disallow: /somepath/*/file/* |
6 | DoppyNL | 5:36 pm Mar 13, 2004 | ||||
| CCGCrawl -- custom crawler/harvester software New crawler software potential for abuse |
2 | privacyman | 1:38 pm Mar 13, 2004 | ||||
| How to exclude bots from specific domains if all point to same place? We have foreign domains that all point to the same folder |
2 | alpine | 6:06 am Mar 13, 2004 | ||||
| Yahoo Images |
2 | RammsteinNicCage | 5:56 am Mar 13, 2004 | ||||
| Who is NaverBot-1.0 (NHN Corp. Just wondering what bot this is.... |
2 | RobbieD | 11:51 am Mar 12, 2004 | ||||
| meta-robots tag; wil link be followed or not? |
2 | DoppyNL | 5:30 am Mar 12, 2004 | ||||
| robots.txt not working Disallows are still being indexed |
3 | jchance | 5:21 am Mar 12, 2004 | ||||
| Can't seem to block Xenu |
6 | WebBender | 12:55 am Mar 12, 2004 | ||||
| php and 'include' files |
3 | charlie12345 | 3:19 am Mar 11, 2004 | ||||
| Weird robot... Google? |
2 | motorhaven | 2:14 am Mar 11, 2004 | ||||
| newbie question |
2 | maxidrom11 | 2:10 am Mar 11, 2004 | ||||
| How To Tell Robots to Index My Site Everyday I need it to come everyday and update my website content everday |
2 | dhaliwal | 9:01 am Mar 9, 2004 | ||||
| Some robot keeps calling a 301 page about 10000 entries till I banned the IP |
1 | walkman | 9:25 pm Mar 8, 2004 |
