indyank - 6:11 am on Sep 3, 2011 (gmt 0) [edited by: indyank at 6:34 am (utc) on Sep 3, 2011]
graeme_p, what is different? Do not mix up facebook with a search engine bot. AFAIK facebook doesn't have any bot to crawl websites or pages on websites like the search robots.
Yes, google does mix up social and search a lot these days to confuse several webmasters like you. However, whenever googlebot crawls a page on a site, it is supposed to check the robots.txt to find what is disallwed and obey the rules there.It is googlebot which is supposed to determine whether a page can be crawled and indexed.
[edited by: indyank at 6:34 am (utc) on Sep 3, 2011]