Welcome to WebmasterWorld Guest from 54.196.153.46

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
How to exclude bots from specific domains if all point to same place?
We have foreign domains that all point to the same folder
2 alpine 6:06 am Mar 13, 2004
Yahoo Images
2 RammsteinNicCage 5:56 am Mar 13, 2004
Who is NaverBot-1.0 (NHN Corp.
Just wondering what bot this is....
2 RobbieD 11:51 am Mar 12, 2004
meta-robots tag; wil link be followed or not?
2 DoppyNL 5:30 am Mar 12, 2004
robots.txt not working
Disallows are still being indexed
3 jchance 5:21 am Mar 12, 2004
Can't seem to block Xenu
6 WebBender 12:55 am Mar 12, 2004
php and 'include' files
3 charlie12345 3:19 am Mar 11, 2004
Weird robot... Google?
2 motorhaven 2:14 am Mar 11, 2004
newbie question
2 maxidrom11 2:10 am Mar 11, 2004
How To Tell Robots to Index My Site Everyday
I need it to come everyday and update my website content everday
2 dhaliwal 9:01 am Mar 9, 2004
Some robot keeps calling a 301 page
about 10000 entries till I banned the IP
1 walkman 9:25 pm Mar 8, 2004
How to Disallow Jeeves?
5 skuba 7:03 pm Mar 5, 2004
Is this done correctly
2 mikemcs 3:00 pm Mar 3, 2004
No Robots.txt page
what's the harm?
7 dauction 4:11 am Mar 2, 2004
Robots.txt files
Who to Disallow
6 Uniterra 5:08 pm Mar 1, 2004
How to disallow doorway pages from indexing?
4 Voyteck 2:30 pm Mar 1, 2004
Directory patterns in query strings
Will robots.txt block them?
2 Wail 8:13 pm Feb 29, 2004
list of bots to ban
it's getting ridiculous but don't want to ban any useful ones
2 walkman 4:48 pm Feb 29, 2004
should the file .htaccess visible
7 newbies 3:44 am Feb 28, 2004
Can I disallow an entire directory but one file?
too many files to list, plus I'd be advertising them
6 walkman 4:13 am Feb 26, 2004
Robots.txt and Index,nofollow
do they have to match
4 graywolf 10:38 pm Feb 22, 2004
New to Robots
New to Robots
7 gotmallz 8:32 pm Feb 22, 2004
hmmm confused
this makes no sense
1 gotmallz 6:15 am Feb 22, 2004
Confirm I've done this right please
9 Essel 5:11 pm Feb 19, 2004
Banning from the entire webserver?
2 Essel 5:57 pm Feb 18, 2004