Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
||3||gettin_there||3:06 pm Dec 1, 2004|
|Crawling of non-linked pages
Over 18's area
|3||theleveller||2:43 pm Dec 1, 2004|
|Ask Jeeves Question
Need to ban a bot
|2||Marcus_Aurelius||10:13 am Dec 1, 2004|
|Webmaster World's Robots.txt
Why don't they allow msnbot?
|6||Rollo||6:11 pm Nov 30, 2004|
|Content of Robots.txt files
You wouldn't believe what some hosts think should be in a Robots.txt file!
|9||AlucardSpyderWriter||8:17 pm Nov 28, 2004|
|How would I disallow Google...
...and allow all others?
|3||Rollo||5:35 pm Nov 28, 2004|
|Robot.txt specialists: will Disallow: /? work?
need to block google from ...com/?tracking.
|1||walkman||7:46 am Nov 28, 2004|
|robots.txt-newbie needs help
Is this correct?
|4||cybervette||5:18 pm Nov 26, 2004|
|Changing your Mind with robots.txt
Will googlebot come back to the party?
|2||Spine||4:41 pm Nov 26, 2004|
||1||apan||12:24 pm Nov 25, 2004|
|robots.txt validator problem?
it is reading my index.htm file?
|2||mbatta||11:34 pm Nov 22, 2004|
|Syntax: User-agent: Mediapartners-Google*
||5||jhubb||3:21 pm Nov 22, 2004|
I asked whether does this, but the engine spider ip address has any to use
|2||ericchui||11:18 am Nov 22, 2004|
|Protecting robots.txt file from bad guys
Is it enough to passwoed protect those dirs?
|3||damlag||4:10 pm Nov 21, 2004|
|How to Exclude from Yahoo/MSN results!
I want to exclude underconstruction site
|2||fm101||1:49 am Nov 21, 2004|
|How to block all subdirectories?
||2||TerryMc||7:05 pm Nov 17, 2004|
|How to exclude dynamic URLS?
So Search engines dont get bogged down.
|7||Tech2004||4:54 am Nov 16, 2004|
|Effect of Robots.txt
||5||seolancer||5:31 am Nov 15, 2004|
|robots.txt - newbie needs help
||5||geezer_uk||5:39 am Nov 10, 2004|
is this right
|2||rbacal||2:59 am Nov 10, 2004|
|Is robot.txt a must for good SEO?
--whats the purpose of robot.txt
|4||redleaf||12:24 am Nov 9, 2004|
|Weird 404 error hitting on robot.txt
anyone experienced this before?
|3||redleaf||9:30 am Nov 8, 2004|
|robots.txt and free web hosting
||3||rhodopsin||7:46 am Nov 8, 2004|
|Possible to have a page disalowed by robots.txt but indexed by adsense
Possible to do this?
|2||brakkar||9:16 pm Nov 5, 2004|
Peerbot or I misunderstood the robot exclusion protocol
|3||WebJoe||9:11 pm Nov 5, 2004|