homepage Welcome to WebmasterWorld Guest from 54.204.68.109
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Would crawlers crawl this?
Usiing the correct robots.txt technique
ulysee




msg:1527642
 6:40 pm on Nov 14, 2003 (gmt 0)

Let's say I have a robots.txt file like this:

User-agent: anybot
Disallow: /redir.php

Would bots be able to follow url's like this:
[domain.com...]
or would it not follow the url above?.

 

dmorison




msg:1527643
 7:31 pm on Nov 14, 2003 (gmt 0)

A robot following robots.txt to the letter would not reach "http://www.anotherdomain.com/" by virtue of requesting redir.php.

However; what the robot might do (and it is believed that Googlebot does this now) is simply add anything that looks like a valid URL to its crawl list; and so "http://www.anotherdomain.com/" would be spotted as a potential crawl target and added to the list.

If you want to make sure that the links are found then don't do what you're proposing; find another way.

ulysee




msg:1527644
 7:51 pm on Nov 14, 2003 (gmt 0)

I want to make sure that links are not found by any crawler, any suggestions?.

dmorison




msg:1527645
 3:02 pm on Nov 15, 2003 (gmt 0)

Gonna be tricky; you could do something clever with JavaScript; but again, the search engines are starting to parse JavaScript and would be able to uncover anything that you are making available as a click-able link to a human.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved