Yes, that will be a problem if the site doesn't respond to "http://domain.com". The port number has to be returned to the spider. Most se's will not spider a site that doesn't initially respond to a request on port 80. Most will strip the port number for the intial query. It will also not store the port number between queries. Using alternative port numbers, used to be one of the real tricks of seo (it's just like multilevel domains).