Welcome to WebmasterWorld Guest from 184.108.40.206
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|stop Yahoo's crawler but for specific domains
how to stop yahoo's crawler for specif domains
|2||nrueda||11:40 pm Nov 8, 2005|
|wildcard for googlebot
||4||fzx5v0||4:02 pm Nov 7, 2005|
|Proposing updates to the robots.txt standard
It is time for a change
|16||volatilegx||9:20 pm Nov 4, 2005|
|what i did wrong?
||5||fluxer||3:45 pm Nov 4, 2005|
|protected ffiles and robots
||2||StickeR||7:48 pm Nov 3, 2005|
|Exculde php url with vars using robots.txt
Exclude urls with vars using robots.txt
|2||alexmart||7:44 pm Nov 3, 2005|
|Is Temporary Disallow a Problem Later?
||6||bumpaw||2:10 pm Nov 3, 2005|
||7||Autoweb||8:25 am Nov 3, 2005|
|Added robots.txt - 3 days later rankings gone!
||11||Block19Row13||3:45 pm Nov 2, 2005|
|OK now I am baffled - question from new to SEO
||4||Iwrite||12:38 pm Nov 2, 2005|
|Robots.text question, disallowing files
||10||mikeD||1:37 am Nov 2, 2005|
|crawl delay in google, msn, yahoo etc.
|4||freegk||7:03 pm Nov 1, 2005|
|Is robots.txt That necessary?
||5||makzan||7:02 pm Nov 1, 2005|
|after google indexs is it too late to use Disallow
||2||watercrazed||8:51 pm Oct 30, 2005|
|Exclude by filename, not directory?
||19||joergnw10||7:36 pm Oct 30, 2005|
|Robots.txt File to Block from Dynamic URLs
Trying to eliminate duplicate content...
|3||BillyS||2:07 am Oct 28, 2005|
|disallowing a folder with its own domain name
subdomain being spidered although it has its own domain name
|2||chandrika||8:52 pm Oct 26, 2005|
|robots.txt in a folder not root
will this work?
|3||fatpeter||6:27 am Oct 26, 2005|
|Robots.txt and Query Strings
Can they block non directory paths?
|3||soquinn||4:45 am Oct 26, 2005|
|Spider and registred access to web
||5||andreabg||8:07 am Oct 19, 2005|
|google accessing blocked image files
why does it appear that google is not abiding robots.txt?
|3||latimer||8:01 am Oct 19, 2005|
|Disallow all Directories
||2||Omega1Megs||8:48 pm Oct 14, 2005|
|Disallow: line in robots.txt
||2||stefan009||5:33 pm Oct 14, 2005|
|Can a robots.txt be too long?
||3||gingerspice||3:32 am Oct 13, 2005|
|Log file is robot.txt file?
Log file is robot.txt file?
|5||i_am_dhaval||9:20 pm Oct 12, 2005|