Just set up the networking on each machine so that they can 'see' each other -- The Network setup wizard will walk you through most of this. On XP, the path is: Start->Control Panel->Network Setup Wizard.
Just as an addition, you can also use a crossover cable connected directly between the two computers. But, if you already have the router, that'd be easier. Or, at least, the cables would be easier to find.