Welcome to WebmasterWorld Guest from 54.167.58.159

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
User-agent: ...
3 gettin_there 3:06 pm Dec 1, 2004
Crawling of non-linked pages
Over 18's area
3 theleveller 2:43 pm Dec 1, 2004
Ask Jeeves Question
Need to ban a bot
2 Marcus_Aurelius 10:13 am Dec 1, 2004
Webmaster World's Robots.txt
Why don't they allow msnbot?
6 Rollo 6:11 pm Nov 30, 2004
Content of Robots.txt files
You wouldn't believe what some hosts think should be in a Robots.txt file!
9 AlucardSpyderWriter 8:17 pm Nov 28, 2004
How would I disallow Google...
...and allow all others?
3 Rollo 5:35 pm Nov 28, 2004
Robot.txt specialists: will Disallow: /? work?
need to block google from ...com/?tracking.
1 walkman 7:46 am Nov 28, 2004
robots.txt-newbie needs help
Is this correct?
4 cybervette 5:18 pm Nov 26, 2004
Changing your Mind with robots.txt
Will googlebot come back to the party?
2 Spine 4:41 pm Nov 26, 2004
dynamic folders
1 apan 12:24 pm Nov 25, 2004
robots.txt validator problem?
it is reading my index.htm file?
2 mbatta 11:34 pm Nov 22, 2004
Syntax: User-agent: Mediapartners-Google*
5 jhubb 3:21 pm Nov 22, 2004
robots.txt question
I asked whether does this, but the engine spider ip address has any to use
2 ericchui 11:18 am Nov 22, 2004
Protecting robots.txt file from bad guys
Is it enough to passwoed protect those dirs?
3 damlag 4:10 pm Nov 21, 2004
How to Exclude from Yahoo/MSN results!
I want to exclude underconstruction site
2 fm101 1:49 am Nov 21, 2004
How to block all subdirectories?
2 TerryMc 7:05 pm Nov 17, 2004
How to exclude dynamic URLS?
So Search engines dont get bogged down.
7 Tech2004 4:54 am Nov 16, 2004
Effect of Robots.txt
5 seolancer 5:31 am Nov 15, 2004
robots.txt - newbie needs help
5 geezer_uk 5:39 am Nov 10, 2004
excluding cgi-bin
is this right
2 rbacal 2:59 am Nov 10, 2004
Is robot.txt a must for good SEO?
--whats the purpose of robot.txt
4 redleaf 12:24 am Nov 9, 2004
Weird 404 error hitting on robot.txt
anyone experienced this before?
3 redleaf 9:30 am Nov 8, 2004
robots.txt and free web hosting
3 rhodopsin 7:46 am Nov 8, 2004
Possible to have a page disalowed by robots.txt but indexed by adsense
Possible to do this?
2 brakkar 9:16 pm Nov 5, 2004
Protocol interpretation
Peerbot or I misunderstood the robot exclusion protocol
3 WebJoe 9:11 pm Nov 5, 2004