Msg#: 4284289 posted 7:38 pm on Mar 19, 2011 (gmt 0)
Hello goodroi, I am hoping that you or someone else can help me with one: using Firefox 4.11 Beta, the Firebug add-on has a feature that not only allows me to examine all of my wordpress.com blogs robots.txt, but also compare them (which are all the same), to high performing wordpress sites and I and another site that is similar to mine, are being restricted greatly, as it appears. Can someone tell me the procedure to change this item in wordpress.com? I copied and pasted all of mine as well as the high performing wordpress.com site. The Firebug is a very good developers tool as well as other Firefox add-ons. I just wish I knew how to use them. I thought that I would change my sites to the same robots.txt that I copied and pasted from the high performing site. Thank you for any and all assistance!
Msg#: 4284289 posted 5:59 pm on Mar 21, 2011 (gmt 0)
Using a robots.txt file from another site is not always a good idea. A robots.txt that is good for Site A is probably not ideal for Site B unless they have both been configured exactly alike and they both have the same objectives. I would encourage you to create a custom robots.txt file to best fit your site.
I am not sure that wordpress.com allows for robots.txt to be changed. I think you might need to be hosting the wordpress on your own site to do that.