Welcome to WebmasterWorld Guest from 50.19.0.90

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt Newbie Question

     
6:33 pm on Nov 30, 2004 (gmt 0)

New User

10+ Year Member

joined:Nov 24, 2004
posts:38
votes: 0


I've been a web developer for years, but am pretty new to the concept of robots.txt. Perhaps somebody can point me to a general thread that explains it, or just answer my simple questions.

As I understand it, robots.txt limits the indexing of your site for whichever searchbots or spambots you mention. Does it also facilitate indexing in some way? If I do not put a robots.txt in my site at all, will my page ranking with some search engines suffer?

I've found a way around email harvesting robots, by using a javascript function to put together an email address. Given that, do I really need any further protection for indexing spiders?

6:36 pm on Nov 30, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member topr8 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 19, 2002
posts:3169
votes: 8


robots.txt only prevents spidering it doesn't enable spidering.

unfortunately there are many bad bots that ignore it, so it doesn't prevent many site rippers and scrapers anyway.

>>If I do not put a robots.txt in my site at all, will my page ranking with some search engines suffer?

imo not at all

6:52 pm on Nov 30, 2004 (gmt 0)

New User

10+ Year Member

joined:Nov 24, 2004
posts:38
votes: 0


Thanks! What I gleaned from your reply is that I don't need to worry myself about this topic. Nice. One less thing.
12:01 am on Dec 2, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member zeus is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 28, 2002
posts:3443
votes: 1


Im introuble with hijackers- redireting sites, can this help, I have never used robot text

User-agent: *
Disallow: /tracker2.php?url=http://www.mydomain.com%2Fmykeyword.2.htm
Disallow: /tracker2.php?url=http://www.mydomain.com%2Fpocket-pc.htm
Disallow: /tracker2.php

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members