Forum Moderators: goodroi

Message Too Old, No Replies

Are there any tricky uses of robot.txt file?

robot.xt file to be indexed by SEs?

         

yuvika

8:38 am on Aug 1, 2006 (gmt 0)

10+ Year Member



One of my frnd suggested me to upload robot.txt file in the root directory, in order to be indexed by SEs. But as far as i know n my search on 'google' and 'webmasterworld' is concerned, robot.txt file is used to prevent SE spiders to index perticular webpages.
So are there any tricky ways to use robot.txt file?

I was also suggested to create navigation for my site using front page. How do i create site navigation for my already existed website, which has been created using another tool?

Any veteran SEO master can suggest effective solution.

AjiNIMC

6:54 pm on Aug 6, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Some people block link pages to cheat link partners.

KenB

7:49 pm on Aug 6, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I lace my robots.txt file with faux URLs that are denied to all robots and then put a bot trap at the URL. Since the URL is only referenced by the robots.txt file I know anything hitting it is a bad bot that isn't playing by the rules and is then black balled from my site.

Is that tricky enough?

Oh I log everything so I can detect any legitimate bots that fall into my trap and exclude them from my ban.

zorde

8:47 am on Aug 22, 2006 (gmt 0)

10+ Year Member



That's clever KenB,

What bad bot's have you come across? I have had a few that really only blow out my stats. Until now I just customise my SQL to not coun't any hits where the referrer is a bot.

Should I be looking something in particular. Should I be paranoid about evil bots?