Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Would crawlers crawl this?

Usiing the correct robots.txt technique

6:40 pm on Nov 14, 2003 (gmt 0)

10+ Year Member

Let's say I have a robots.txt file like this:

User-agent: anybot
Disallow: /redir.php

Would bots be able to follow url's like this:
or would it not follow the url above?.

7:31 pm on Nov 14, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

A robot following robots.txt to the letter would not reach "http://www.anotherdomain.com/" by virtue of requesting redir.php.

However; what the robot might do (and it is believed that Googlebot does this now) is simply add anything that looks like a valid URL to its crawl list; and so "http://www.anotherdomain.com/" would be spotted as a potential crawl target and added to the list.

If you want to make sure that the links are found then don't do what you're proposing; find another way.

7:51 pm on Nov 14, 2003 (gmt 0)

10+ Year Member

I want to make sure that links are not found by any crawler, any suggestions?.
3:02 pm on Nov 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Gonna be tricky; you could do something clever with JavaScript; but again, the search engines are starting to parse JavaScript and would be able to uncover anything that you are making available as a click-able link to a human.

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved