Welcome to WebmasterWorld Guest from 54.167.22.37

Forum Moderators: goodroi

Message Too Old, No Replies

Would crawlers crawl this?

Usiing the correct robots.txt technique

     

ulysee

6:40 pm on Nov 14, 2003 (gmt 0)

10+ Year Member



Let's say I have a robots.txt file like this:

User-agent: anybot
Disallow: /redir.php

Would bots be able to follow url's like this:
[domain.com...]
or would it not follow the url above?.

dmorison

7:31 pm on Nov 14, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



A robot following robots.txt to the letter would not reach "http://www.anotherdomain.com/" by virtue of requesting redir.php.

However; what the robot might do (and it is believed that Googlebot does this now) is simply add anything that looks like a valid URL to its crawl list; and so "http://www.anotherdomain.com/" would be spotted as a potential crawl target and added to the list.

If you want to make sure that the links are found then don't do what you're proposing; find another way.

ulysee

7:51 pm on Nov 14, 2003 (gmt 0)

10+ Year Member



I want to make sure that links are not found by any crawler, any suggestions?.

dmorison

3:02 pm on Nov 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Gonna be tricky; you could do something clever with JavaScript; but again, the search engines are starting to parse JavaScript and would be able to uncover anything that you are making available as a click-able link to a human.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month