| Robots.txt Newbie Question
|
paulpsd
msg:1526357 | 6:33 pm on Nov 30, 2004 (gmt 0) |
I've been a web developer for years, but am pretty new to the concept of robots.txt. Perhaps somebody can point me to a general thread that explains it, or just answer my simple questions. As I understand it, robots.txt limits the indexing of your site for whichever searchbots or spambots you mention. Does it also facilitate indexing in some way? If I do not put a robots.txt in my site at all, will my page ranking with some search engines suffer? I've found a way around email harvesting robots, by using a javascript function to put together an email address. Given that, do I really need any further protection for indexing spiders?
|
topr8
msg:1526358 | 6:36 pm on Nov 30, 2004 (gmt 0) |
robots.txt only prevents spidering it doesn't enable spidering. unfortunately there are many bad bots that ignore it, so it doesn't prevent many site rippers and scrapers anyway. >>If I do not put a robots.txt in my site at all, will my page ranking with some search engines suffer? imo not at all
|
paulpsd
msg:1526359 | 6:52 pm on Nov 30, 2004 (gmt 0) |
Thanks! What I gleaned from your reply is that I don't need to worry myself about this topic. Nice. One less thing.
|
zeus
msg:1526360 | 12:01 am on Dec 2, 2004 (gmt 0) |
Im introuble with hijackers- redireting sites, can this help, I have never used robot text User-agent: *
Disallow: /tracker2.php?url=http://www.mydomain.com%2Fmykeyword.2.htm
Disallow: /tracker2.php?url=http://www.mydomain.com%2Fpocket-pc.htm
Disallow: /tracker2.php
|
|
|
