Welcome to WebmasterWorld Guest from 126.96.36.199
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Keeping spiders off co-branded sites
how to keep spiders from indexing our partner / co-branded sites
|4||BillingOrchard||9:41 pm Dec 23, 2004|
|About robots.txt file
Confusion in robots.txt file...i need hep..
|7||venuc||5:32 am Dec 21, 2004|
How do you tell the real googlebot from spoofers?
|3||teaperson||6:29 pm Dec 20, 2004|
|Tricky Robots Txt Help,
Two sites one robot
|3||badboybart||2:50 pm Dec 20, 2004|
|redirect without confusing spiders
||1||houseofsecrets||4:55 pm Dec 13, 2004|
|Robot.txt file deleted by accident what now?
||2||Hoppo||11:15 pm Dec 12, 2004|
|robots.txt and meta tags
can I use both?
|2||kdawg||11:23 pm Dec 11, 2004|
|Oddly long robots.txt file
Seeking explanation for weird robots.txt file
|7||bendede||9:31 pm Dec 9, 2004|
|4||Kasey22||8:44 pm Dec 7, 2004|
Is this allowed?
|3||maha||5:02 am Dec 6, 2004|
|How do I create two robots.txt for the same directory?
.com and .net are for two different engines but point at the same folder
|3||walkman||9:01 pm Dec 2, 2004|
|Robots.txt Newbie Question
||4||paulpsd||12:01 am Dec 2, 2004|
||3||tomasz||9:31 pm Dec 1, 2004|
|subdomains and robots.txt
||2||gettin_there||8:54 pm Dec 1, 2004|
||3||gettin_there||3:06 pm Dec 1, 2004|
|Crawling of non-linked pages
Over 18's area
|3||theleveller||2:43 pm Dec 1, 2004|
|Ask Jeeves Question
Need to ban a bot
|2||Marcus_Aurelius||10:13 am Dec 1, 2004|
|Webmaster World's Robots.txt
Why don't they allow msnbot?
|6||Rollo||6:11 pm Nov 30, 2004|
|Content of Robots.txt files
You wouldn't believe what some hosts think should be in a Robots.txt file!
|9||AlucardSpyderWriter||8:17 pm Nov 28, 2004|
|How would I disallow Google...
...and allow all others?
|3||Rollo||5:35 pm Nov 28, 2004|
|Robot.txt specialists: will Disallow: /? work?
need to block google from ...com/?tracking.
|1||walkman||7:46 am Nov 28, 2004|
|robots.txt-newbie needs help
Is this correct?
|4||cybervette||5:18 pm Nov 26, 2004|
|Changing your Mind with robots.txt
Will googlebot come back to the party?
|2||Spine||4:41 pm Nov 26, 2004|
||1||apan||12:24 pm Nov 25, 2004|
|robots.txt validator problem?
it is reading my index.htm file?
|2||mbatta||11:34 pm Nov 22, 2004|