It'll help me to explain things if I define some words for our purpose:
A "page" is something you'd load in a browser, like a layout with purty graphics and CSS, links, text, a logo, and some things you can click on. Pages are usually created using HTML and other ingredients. Web sites are usually made of pages. Your MySpace profile is a web page. Pages are usually (almost always) built using HTML, but the Page as a whole also includes all the images and scripts and stylesheets that make it look the way it does.
An XML file, or a URL which serves XML, is usually called a "Document". Don't worry why. But when I refer to an XML Document, I won't mean the same things as a "page".
One ingredient of a Page is the HTML Document. The Document is the DOCTYPE declaration (optional), the <html> tag (required), and everything inside that. In addition to the Document, all the extra stuff like images and stylesheets and scripts comprise a Page.
Both HTML and XML are constructed as Documents. Indeed there's a flavour of HTML, called XHTML, which *is* XML. As mentioned already, XML is a language which lets you create other languages... that's an awkward concept to grasp, so consider this analogy: LEGO is a toy which lets you create other toys. If XHTML is your assembled Millenium Falcon sculpture, XML is the blocks that let you create it. XML has its own rules for validity and syntax, just like LEGO has its own rules for how the blocks snap together. With LEGO you can build the Millenium Falcon, Hogwarts Castle, and a motorized truck. With XML, you can build HTML, RSS, MathML, XSLT, SVG, and dozens of other well-defined languages. All those languages have their own words and grammar, but they all obey the fundamental rules which we define as XML.
Unlike HTML, XML is primarily intended to be consumed by machines. The elements in an XML Document are named and nested in such a way that they are human-readable, but can also be read ("parsed") by a computer program to get the data defined within it.
An XML sitemap is not (usually) intended to be viewed as a Page. It's intended to be read by search engine robots, so they can easily find the pages in your site, request them, index them, and get your site ranking in their Search Engine. Put your XML sitemap in a place where they'll find it - more information about how to do that is available here [sitemaps.org].
When you view an XML Document in a web browser, it looks awful. Most browsers render it as a tree with nodes you can expand or collapse. The browser makes it readable, and useful, but it doesn't exude much character.
Here is an XML file that gets used a lot in online tutorials:
When you view your sitemap.xml in a browser, it should look like that.
There is a way to make an XML Document look more like a Page - and that is by defining a stylesheet for it written in XSLT*, and prettified with CSS. Here is an example of an XML Document [webmasterworld.com] that is being rendered in your browser like it's a Page. In fact, it's fair to call that an XML Page. If you View the Source of that page, you'll see that it's not HTML, it's XML.
* ( XSLT has notoriously steep learning curve, but those who pursue mastery of XSLT tend to find it's an indispensible part of a Webmaster's skillset. It's prerequisite to learn the basics of XML before trying XSLT. )
On many sites I've built, there are two places where you can find a site map:
http://www.example.com/sitemap.xml <- this is the one put there for Google and its robotic friends
http://www.example.com/sitemap.html <- this is the one for you. There's a link to it in the layout footer.
So, to answer your question (finally), yes you can link to an XML Document, and you can even link to an XML Page. Entire sites can be built in XML instead of HTML. However, "normal" pages don't usually do that.
Links to XML Documents are quite common - whenever you see an RSS feed button, that's a link to an XML Document. (RSS is one of those languages built with XML)
XML is a very rich subject, and it's used for lots of other things besides making sitemaps. go and explore!