Welcome to WebmasterWorld Guest from 54.221.9.209

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Disallowing https
Can this be done via robots.txt?
2 yosmc 8:45 pm Apr 9, 2005
mod_rewrite and robots.txt
General discussion on use mod_rewrite with robots.txt
7 Vaddy 4:34 pm Apr 7, 2005
New moderator for robots.txt forum
4 engine 7:15 pm Apr 5, 2005
Subdomains and restricted access Web pages
Search Engine Optimization
2 anshul 7:06 pm Apr 5, 2005
What is: "Googles robots NoCache tag" please?
4 beyondenchanted 8:37 pm Apr 4, 2005
Robot.txt
Where it goes..
3 oresam 5:06 pm Apr 4, 2005
Banning googlebot from individual pages via robots.txt
Meta tags are just not doing the job properly
1 Vimes 8:07 am Mar 31, 2005
Wildcards in robots.txt
2 madmatt69 12:52 am Mar 31, 2005
Blocking all files with a .ASP extension
Is this possible with Robots.txt
1 marketrevolution 6:21 pm Mar 30, 2005
disallowing by script variable
6 wheel 6:30 am Mar 29, 2005
Syntax help please
2 jcambert 3:14 am Mar 29, 2005
Specifically allow one bot to index a page
please check my code ;)
5 RonPK 10:37 am Mar 27, 2005
Bots using robots
Do they really have to?
5 cooldoug 8:54 pm Mar 26, 2005
Image disallow
2 Dexie 1:20 pm Mar 26, 2005
Robots.txt Question
Will this cover it?
4 rover 5:40 pm Mar 25, 2005
Robot.txt woes with my mac
Trouble with robot.txt
4 mkfoto 2:21 am Mar 22, 2005
web logs Primer
Looking for general help/information interpreting web site logs
1 scrgeek 3:05 pm Mar 21, 2005
Robots.txt
Disallowed pages listed
4 unperturbed 7:28 pm Mar 20, 2005
robots.txt confusion
question about where to put robots.txt
10 SplitPersona 3:13 pm Mar 17, 2005
exclude vb with robots
1 johndk 9:55 am Mar 16, 2005
Filenames have spaces - How to reflect in robots.txt
Website has filenames that contain spaces. What to do?
4 SanDiego_Art 2:54 am Mar 16, 2005
restriction for a certain country
is it possible to do that?
3 lucky 8:06 pm Mar 15, 2005
need help blocking a site
not sure if this is possible
11 willeffects 5:23 am Mar 15, 2005
PriceGrabber UserAgent
1 driwashsolutions 1:59 am Mar 15, 2005
crawl confusion
Understanding my hits log
5 cardpro 8:59 am Mar 14, 2005