Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|CCGCrawl -- custom crawler/harvester software
New crawler software potential for abuse
|2||privacyman||1:38 pm Mar 13, 2004|
|How to exclude bots from specific domains if all point to same place?
We have foreign domains that all point to the same folder
|2||alpine||6:06 am Mar 13, 2004|
||2||RammsteinNicCage||5:56 am Mar 13, 2004|
|Who is NaverBot-1.0 (NHN Corp.
Just wondering what bot this is....
|2||RobbieD||11:51 am Mar 12, 2004|
|meta-robots tag; wil link be followed or not?
||2||DoppyNL||5:30 am Mar 12, 2004|
|robots.txt not working
Disallows are still being indexed
|3||jchance||5:21 am Mar 12, 2004|
|Can't seem to block Xenu
||6||WebBender||12:55 am Mar 12, 2004|
|php and 'include' files
||3||charlie12345||3:19 am Mar 11, 2004|
|Weird robot... Google?
||2||motorhaven||2:14 am Mar 11, 2004|
||2||maxidrom11||2:10 am Mar 11, 2004|
|How To Tell Robots to Index My Site Everyday
I need it to come everyday and update my website content everday
|2||dhaliwal||9:01 am Mar 9, 2004|
|Some robot keeps calling a 301 page
about 10000 entries till I banned the IP
|1||walkman||9:25 pm Mar 8, 2004|
|How to Disallow Jeeves?
||5||skuba||7:03 pm Mar 5, 2004|
|Is this done correctly
||2||mikemcs||3:00 pm Mar 3, 2004|
|No Robots.txt page
what's the harm?
|7||dauction||4:11 am Mar 2, 2004|
Who to Disallow
|6||Uniterra||5:08 pm Mar 1, 2004|
|How to disallow doorway pages from indexing?
||4||Voyteck||2:30 pm Mar 1, 2004|
|Directory patterns in query strings
Will robots.txt block them?
|2||Wail||8:13 pm Feb 29, 2004|
|list of bots to ban
it's getting ridiculous but don't want to ban any useful ones
|2||walkman||4:48 pm Feb 29, 2004|
|should the file .htaccess visible
||7||newbies||3:44 am Feb 28, 2004|
|Can I disallow an entire directory but one file?
too many files to list, plus I'd be advertising them
|6||walkman||4:13 am Feb 26, 2004|
|Robots.txt and Index,nofollow
do they have to match
|4||graywolf||10:38 pm Feb 22, 2004|
|New to Robots
New to Robots
|7||gotmallz||8:32 pm Feb 22, 2004|
this makes no sense
|1||gotmallz||6:15 am Feb 22, 2004|
|Confirm I've done this right please
||9||Essel||5:11 pm Feb 19, 2004|