Welcome to WebmasterWorld Guest from 54.234.129.215

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Content of Robots.txt files
You wouldn't believe what some hosts think should be in a Robots.txt file!
9 AlucardSpyderWriter 8:17 pm Nov 28, 2004
How would I disallow Google...
...and allow all others?
3 Rollo 5:35 pm Nov 28, 2004
Robot.txt specialists: will Disallow: /? work?
need to block google from ...com/?tracking.
1 walkman 7:46 am Nov 28, 2004
robots.txt-newbie needs help
Is this correct?
4 cybervette 5:18 pm Nov 26, 2004
Changing your Mind with robots.txt
Will googlebot come back to the party?
2 Spine 4:41 pm Nov 26, 2004
dynamic folders
1 apan 12:24 pm Nov 25, 2004
robots.txt validator problem?
it is reading my index.htm file?
2 mbatta 11:34 pm Nov 22, 2004
Syntax: User-agent: Mediapartners-Google*
5 jhubb 3:21 pm Nov 22, 2004
robots.txt question
I asked whether does this, but the engine spider ip address has any to use
2 ericchui 11:18 am Nov 22, 2004
Protecting robots.txt file from bad guys
Is it enough to passwoed protect those dirs?
3 damlag 4:10 pm Nov 21, 2004
How to Exclude from Yahoo/MSN results!
I want to exclude underconstruction site
2 fm101 1:49 am Nov 21, 2004
How to block all subdirectories?
2 TerryMc 7:05 pm Nov 17, 2004
How to exclude dynamic URLS?
So Search engines dont get bogged down.
7 Tech2004 4:54 am Nov 16, 2004
Effect of Robots.txt
5 seolancer 5:31 am Nov 15, 2004
robots.txt - newbie needs help
5 geezer_uk 5:39 am Nov 10, 2004
excluding cgi-bin
is this right
2 rbacal 2:59 am Nov 10, 2004
Is robot.txt a must for good SEO?
--whats the purpose of robot.txt
4 redleaf 12:24 am Nov 9, 2004
Weird 404 error hitting on robot.txt
anyone experienced this before?
3 redleaf 9:30 am Nov 8, 2004
robots.txt and free web hosting
3 rhodopsin 7:46 am Nov 8, 2004
Possible to have a page disalowed by robots.txt but indexed by adsense
Possible to do this?
2 brakkar 9:16 pm Nov 5, 2004
Protocol interpretation
Peerbot or I misunderstood the robot exclusion protocol
3 WebJoe 9:11 pm Nov 5, 2004
Need help about File Robots.txt
how do I know spiders visit to my site or not?
3 viva 10:15 am Nov 4, 2004
what happen with robot exclusion and iframes?
could it be a problem to exclude content that appears inside a home iframe?
2 mgorricho 5:03 pm Nov 3, 2004
protocol extension resource control
proposed extension to robots.txt protocol allowing resource control
1 copsewood 11:29 am Nov 2, 2004
Querystring check
2 papamaku 9:36 am Nov 2, 2004

Featured Threads

Hot Threads This Week

Hot Threads This Month