Welcome to WebmasterWorld Guest from 220.127.116.11
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Denying access to everything but the home page?
Using robots.txt to only allow robot access to the home page
|3||monkeylytics||8:00 pm Jan 28, 2008|
|Control Robots with Meta or robots.txt
||3||driller41||12:21 pm Jan 15, 2008|
||12||g1smd||12:18 am Jan 15, 2008|
|robots.txt with wildcard?
||3||tranceformer||12:39 am Jan 13, 2008|
|Disallowing with robots.txt
robots.txt, disallow certain urls
|3||jim_knopf||12:36 am Jan 13, 2008|
|Getting correct user agent
getting from access log the correct user agent for robots.txt
|2||hessodreamy||9:03 pm Jan 11, 2008|
|Scrapers checking robots.txt
Spiders arrived on heels of robots.txt deployment
|4||timster||1:29 am Jan 4, 2008|
|404 Errors in Webmaster Console
What to do with these pages?
|2||rowtc2||11:35 pm Jan 3, 2008|
|directory and subdirectory access
allowing access to subdirectory but not main directory
|3||Mr_Cool_Dude||6:11 pm Jan 2, 2008|
|2||jim_knopf||5:43 pm Jan 2, 2008|
|Reference robots.txt file?
Does anyone have one?
|3||mmiller||11:30 pm Dec 25, 2007|
|how to manage cname in robot
don't was to be indexed
|4||experienced||2:10 pm Dec 21, 2007|
|disallow vs ban in .htaccess
||12||joergnw10||11:10 am Dec 16, 2007|
|How do I prevent the SEs from crawling any links to my site w/HTTPS
I don't want the crawlers to index anything w/ HTTPS protocol
|3||ZydoSEO||8:37 pm Dec 9, 2007|
|yahoo google msn etc.
|1||ComputerHermit||5:56 pm Dec 7, 2007|
|Proposal for robots.txt To Have Greater Flexibility
||12||engine||4:58 pm Nov 30, 2007|
|set crawl rate permanently to faster
||1||agushalim||5:39 pm Nov 28, 2007|
|Wildcards on robots.txt disallows
||11||MindTwist||5:59 pm Nov 27, 2007|
|Site title and description missing in search results
I had made some changes in robots.txt file
|2||ecmedia||2:12 pm Nov 27, 2007|
|perfect robot text files
good robot text files
|2||likwidholdings||12:57 am Nov 27, 2007|
|Google showing site despite blocking in robots.txt
The page is not cached though
|3||ecmedia||4:28 pm Nov 20, 2007|
|Disallow all except adsense
||6||thegreatpretender||4:40 pm Nov 19, 2007|
|How to avoid double indexing
||3||mavi||4:43 pm Nov 13, 2007|
||5||smartcard||10:44 am Nov 13, 2007|
|Robot from colo is on behalf of Google?!
Colo ISP claims robot crawling my site is from 3rd party Google contractor
|5||yodokame||4:03 am Nov 12, 2007|