Welcome to WebmasterWorld Guest from 54.158.166.6

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt Newbie Question

   
6:33 pm on Nov 30, 2004 (gmt 0)

10+ Year Member



I've been a web developer for years, but am pretty new to the concept of robots.txt. Perhaps somebody can point me to a general thread that explains it, or just answer my simple questions.

As I understand it, robots.txt limits the indexing of your site for whichever searchbots or spambots you mention. Does it also facilitate indexing in some way? If I do not put a robots.txt in my site at all, will my page ranking with some search engines suffer?

I've found a way around email harvesting robots, by using a javascript function to put together an email address. Given that, do I really need any further protection for indexing spiders?

6:36 pm on Nov 30, 2004 (gmt 0)

WebmasterWorld Senior Member topr8 is a WebmasterWorld Top Contributor of All Time 10+ Year Member



robots.txt only prevents spidering it doesn't enable spidering.

unfortunately there are many bad bots that ignore it, so it doesn't prevent many site rippers and scrapers anyway.

>>If I do not put a robots.txt in my site at all, will my page ranking with some search engines suffer?

imo not at all

6:52 pm on Nov 30, 2004 (gmt 0)

10+ Year Member



Thanks! What I gleaned from your reply is that I don't need to worry myself about this topic. Nice. One less thing.
12:01 am on Dec 2, 2004 (gmt 0)

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Im introuble with hijackers- redireting sites, can this help, I have never used robot text

User-agent: *
Disallow: /tracker2.php?url=http://www.mydomain.com%2Fmykeyword.2.htm
Disallow: /tracker2.php?url=http://www.mydomain.com%2Fpocket-pc.htm
Disallow: /tracker2.php