Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
Is this allowed?
|3||maha||5:02 am Dec 6, 2004|
|How do I create two robots.txt for the same directory?
.com and .net are for two different engines but point at the same folder
|3||walkman||9:01 pm Dec 2, 2004|
|Robots.txt Newbie Question
||4||paulpsd||12:01 am Dec 2, 2004|
||3||tomasz||9:31 pm Dec 1, 2004|
|subdomains and robots.txt
||2||gettin_there||8:54 pm Dec 1, 2004|
||3||gettin_there||3:06 pm Dec 1, 2004|
|Crawling of non-linked pages
Over 18's area
|3||theleveller||2:43 pm Dec 1, 2004|
|Ask Jeeves Question
Need to ban a bot
|2||Marcus_Aurelius||10:13 am Dec 1, 2004|
|Webmaster World's Robots.txt
Why don't they allow msnbot?
|6||Rollo||6:11 pm Nov 30, 2004|
|Content of Robots.txt files
You wouldn't believe what some hosts think should be in a Robots.txt file!
|9||AlucardSpyderWriter||8:17 pm Nov 28, 2004|
|How would I disallow Google...
...and allow all others?
|3||Rollo||5:35 pm Nov 28, 2004|
|Robot.txt specialists: will Disallow: /? work?
need to block google from ...com/?tracking.
|1||walkman||7:46 am Nov 28, 2004|
|robots.txt-newbie needs help
Is this correct?
|4||cybervette||5:18 pm Nov 26, 2004|
|Changing your Mind with robots.txt
Will googlebot come back to the party?
|2||Spine||4:41 pm Nov 26, 2004|
||1||apan||12:24 pm Nov 25, 2004|
|robots.txt validator problem?
it is reading my index.htm file?
|2||mbatta||11:34 pm Nov 22, 2004|
|Syntax: User-agent: Mediapartners-Google*
||5||jhubb||3:21 pm Nov 22, 2004|
I asked whether does this, but the engine spider ip address has any to use
|2||ericchui||11:18 am Nov 22, 2004|
|Protecting robots.txt file from bad guys
Is it enough to passwoed protect those dirs?
|3||damlag||4:10 pm Nov 21, 2004|
|How to Exclude from Yahoo/MSN results!
I want to exclude underconstruction site
|2||fm101||1:49 am Nov 21, 2004|
|How to block all subdirectories?
||2||TerryMc||7:05 pm Nov 17, 2004|
|How to exclude dynamic URLS?
So Search engines dont get bogged down.
|7||Tech2004||4:54 am Nov 16, 2004|
|Effect of Robots.txt
||5||seolancer||5:31 am Nov 15, 2004|
|robots.txt - newbie needs help
||5||geezer_uk||5:39 am Nov 10, 2004|
is this right
|2||rbacal||2:59 am Nov 10, 2004|