Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Not sure how to handle space character
||5||Jon_King||1:48 am Jan 6, 2005|
|Problem of syntax and ordering
||1||esllou||11:50 am Jan 4, 2005|
|Will robots.txt affect IE or other browsers?
A little confuse...
|2||iProgram||4:34 am Jan 4, 2005|
|what are the names of the image bots?
I can't ban them if I don't know their names
|8||stapel||5:17 pm Jan 2, 2005|
|Blocking images folder from spiders
Good or bad idea?
|2||twist||10:17 pm Jan 1, 2005|
|Sudden Increase in robots.txt Hits
||3||Rob_Moore||10:04 pm Dec 26, 2004|
|Keeping spiders off co-branded sites
how to keep spiders from indexing our partner / co-branded sites
|4||BillingOrchard||9:41 pm Dec 23, 2004|
|About robots.txt file
Confusion in robots.txt file...i need hep..
|7||venuc||5:32 am Dec 21, 2004|
How do you tell the real googlebot from spoofers?
|3||teaperson||6:29 pm Dec 20, 2004|
|Tricky Robots Txt Help,
Two sites one robot
|3||badboybart||2:50 pm Dec 20, 2004|
|redirect without confusing spiders
||1||houseofsecrets||4:55 pm Dec 13, 2004|
|Robot.txt file deleted by accident what now?
||2||Hoppo||11:15 pm Dec 12, 2004|
|robots.txt and meta tags
can I use both?
|2||kdawg||11:23 pm Dec 11, 2004|
|Oddly long robots.txt file
Seeking explanation for weird robots.txt file
|7||bendede||9:31 pm Dec 9, 2004|
|4||Kasey22||8:44 pm Dec 7, 2004|
Is this allowed?
|3||maha||5:02 am Dec 6, 2004|
|How do I create two robots.txt for the same directory?
.com and .net are for two different engines but point at the same folder
|3||walkman||9:01 pm Dec 2, 2004|
|Robots.txt Newbie Question
||4||paulpsd||12:01 am Dec 2, 2004|
||3||tomasz||9:31 pm Dec 1, 2004|
|subdomains and robots.txt
||2||gettin_there||8:54 pm Dec 1, 2004|
||3||gettin_there||3:06 pm Dec 1, 2004|
|Crawling of non-linked pages
Over 18's area
|3||theleveller||2:43 pm Dec 1, 2004|
|Ask Jeeves Question
Need to ban a bot
|2||Marcus_Aurelius||10:13 am Dec 1, 2004|
|Webmaster World's Robots.txt
Why don't they allow msnbot?
|6||Rollo||6:11 pm Nov 30, 2004|
|Content of Robots.txt files
You wouldn't believe what some hosts think should be in a Robots.txt file!
|9||AlucardSpyderWriter||8:17 pm Nov 28, 2004|