Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Disallow all except adsense
||6||thegreatpretender||4:40 pm Nov 19, 2007|
|How to avoid double indexing
||3||mavi||4:43 pm Nov 13, 2007|
||5||smartcard||10:44 am Nov 13, 2007|
|Robot from colo is on behalf of Google?!
Colo ISP claims robot crawling my site is from 3rd party Google contractor
|5||yodokame||4:03 am Nov 12, 2007|
how it affects the page rank of any website
|2||monikagaba||6:21 pm Nov 1, 2007|
|Exclude ALL bots but major search engines
Trying to make the perfect robots.txt
|5||serpmaster||12:03 pm Oct 25, 2007|
|blocking all except root (index.php)
would disallow ALL , allow index work?
|1||bkode||7:56 am Oct 24, 2007|
|http code 200 mystery
is the page served or not!?
|3||dailyclicker||5:14 pm Oct 13, 2007|
|Do robots get cookies?
robots cookies possible dumb question
|4||Michel_Samuel||10:55 am Oct 11, 2007|
excluding multiple root directories.
|5||Michel_Samuel||10:05 pm Oct 10, 2007|
|Multiple robots.txt files in different directories -will it work?
placing a robots.txt file in sub directories.
|3||Michel_Samuel||7:44 pm Oct 9, 2007|
|Keep physical hosting domain from being indexed?
||5||php4U||2:21 am Oct 9, 2007|
|How do I not allow google
||2||pizzaguy||10:31 pm Oct 3, 2007|
|Should this file be "Public"
Is it good practice to have the robots.txt file.......
|4||Propools||11:58 am Sep 26, 2007|
|Proper URL & File Structure
Fighting with Robots. Txt
|3||srkmodi||12:34 pm Sep 24, 2007|
|Use & or & amp; in robots.txt
Which to use?
|3||g1smd||1:53 pm Sep 21, 2007|
|Using allow to index search-only pages
||3||timmbbo||12:31 pm Sep 18, 2007|
|Disallow - wait a day - add pages
||1||brevetoxin||1:36 am Sep 12, 2007|
|Robots.txt to improve ranks
Will blocking of internal pages through robots.txt improve SE rankings
|3||pro_seo||5:18 am Sep 10, 2007|
|Robots INclusions standard?
||13||mikomido||12:28 am Sep 9, 2007|
|blocking the hostsite but not the hosted sites
||3||messemaeker||12:12 pm Sep 7, 2007|
|Blocking Windows Live robot/spider.
||2||Tidal2||10:56 am Sep 7, 2007|
|Do we have to use robots.txt on every website?
for robots.txt issues
|5||masood82||8:42 am Sep 6, 2007|
|How to block tags?
hundreds of undesired links !
|3||rashe18||4:33 pm Sep 4, 2007|
|Blocked through robots.txt . will the page be crawled ever
||5||getxb||6:02 am Aug 31, 2007|