Welcome to WebmasterWorld Guest from 54.146.239.96

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt and mod_rewrite
just double checking
6 hannamyluv 2:24 am Mar 14, 2004
Several robots.txt?
2 brakkar 1:18 am Mar 14, 2004
Where is the "Top Level"?
is this important these days?
4 chewy 9:46 pm Mar 13, 2004
Does this work?
Disallow: /somepath/*/file/*
6 DoppyNL 5:36 pm Mar 13, 2004
CCGCrawl -- custom crawler/harvester software
New crawler software potential for abuse
2 privacyman 1:38 pm Mar 13, 2004
How to exclude bots from specific domains if all point to same place?
We have foreign domains that all point to the same folder
2 alpine 6:06 am Mar 13, 2004
Yahoo Images
2 RammsteinNicCage 5:56 am Mar 13, 2004
Who is NaverBot-1.0 (NHN Corp.
Just wondering what bot this is....
2 RobbieD 11:51 am Mar 12, 2004
meta-robots tag; wil link be followed or not?
2 DoppyNL 5:30 am Mar 12, 2004
robots.txt not working
Disallows are still being indexed
3 jchance 5:21 am Mar 12, 2004
Can't seem to block Xenu
6 WebBender 12:55 am Mar 12, 2004
php and 'include' files
3 charlie12345 3:19 am Mar 11, 2004
Weird robot... Google?
2 motorhaven 2:14 am Mar 11, 2004
newbie question
2 maxidrom11 2:10 am Mar 11, 2004
How To Tell Robots to Index My Site Everyday
I need it to come everyday and update my website content everday
2 dhaliwal 9:01 am Mar 9, 2004
Some robot keeps calling a 301 page
about 10000 entries till I banned the IP
1 walkman 9:25 pm Mar 8, 2004
How to Disallow Jeeves?
5 skuba 7:03 pm Mar 5, 2004
Is this done correctly
2 mikemcs 3:00 pm Mar 3, 2004
No Robots.txt page
what's the harm?
7 dauction 4:11 am Mar 2, 2004
Robots.txt files
Who to Disallow
6 Uniterra 5:08 pm Mar 1, 2004
How to disallow doorway pages from indexing?
4 Voyteck 2:30 pm Mar 1, 2004
Directory patterns in query strings
Will robots.txt block them?
2 Wail 8:13 pm Feb 29, 2004
list of bots to ban
it's getting ridiculous but don't want to ban any useful ones
2 walkman 4:48 pm Feb 29, 2004
should the file .htaccess visible
7 newbies 3:44 am Feb 28, 2004
Can I disallow an entire directory but one file?
too many files to list, plus I'd be advertising them
6 walkman 4:13 am Feb 26, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month