Welcome to WebmasterWorld Guest from 54.159.240.93

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
robots.txt file
3 ktd121 3:14 am Jul 4, 2003
meta name = robot
What does that do?
7 aroye 5:57 pm Jun 30, 2003
Scooter 3.3 ignores robots.txt?
this is ridiculous
6 berli 2:39 am Jun 27, 2003
Robots.txt
Help please mine is missing?
3 peteredmonds 8:37 pm Jun 25, 2003
Robots.txt
Huh?
2 zoobie 6:52 pm Jun 19, 2003
"robots"
What should it be set at?
4 kimderella86 6:22 pm Jun 18, 2003
Robots.txt exclusion for dynamic page
2 rogerd 9:22 pm Jun 17, 2003
Robots.txt & How your site is spidered
Positives of Robots.txt file aside from disallowing bots
4 floridadesigns 11:31 pm Jun 14, 2003
Why should I obey robots.txt
Developer questions
22 PWalker 1:01 am Jun 13, 2003
Php robots.txt parser script
does that exitst? if not regex help needed.
5 ikbenhet1 2:57 pm Jun 9, 2003
files requested by robots.txt
i would like to find out about the files requested by robots
8 andrea_edwards 7:53 pm Jun 8, 2003
Toolbar and Robots.txt
8 rintrah 8:41 pm Jun 5, 2003
Robots.txt - files in folders
3 rintrah 9:43 pm Jun 4, 2003
robots.txt
bad spider
6 copongcopong 9:11 pm Jun 4, 2003
Robots.txt - What's the Point?
7 rintrah 6:59 pm May 30, 2003
Pragma & Cache-Control
Are these Safe with Search Engines
9 stuart 8:32 pm May 27, 2003
Quick robots.txt question
Do I just need to put the file name
2 bateman_ap 8:21 pm May 24, 2003
Using Allow: / in robots.txt
Google says to use it?
5 werty 3:28 am May 20, 2003
How often should a robot check the robots.txt file?
Your opinions please
8 jrobbio 2:25 am May 18, 2003
Using Robot.txt to Block Spidering of Link pages
Is this a common underhanded practice?
12 jdancing 4:26 pm May 8, 2003
robots.txt
should i even bother if i want the whole site indexed?
9 madeonmoon 4:37 pm May 7, 2003
robots.txt
used for site with 2 urls
3 spinweb 2:11 am May 7, 2003
robots.txt validator warning suggestion
Missing newlines can cause problems, but in some cases can be detected.
2 mortenf 4:42 pm Apr 25, 2003
Dangers of the robots.txt file
A warning to the amateurs like myself
3 joeuz 6:37 am Apr 23, 2003
Robots.txt
Stopping robots visiting single dynamic page.
2 donovana 2:02 am Apr 22, 2003

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month