It can be surprisingly difficult. If the spacecraft is travelling far from planets, then normal orbit calculations should be fine. It's the interactions with planets and moons that are trickier.
There will be a variety of methods, but the space agencies themselves will use powerful computers to calculate trajectories using an iterative step process, each step calculating a position a small time fraction after the previous. All the bodies of the solar system can be taken into account, and the maths will use something like Legendre polynomials to calculate the gravitational potentials as perturbences of an elliptical orbit.
When planetary probes are being planned, they will run and re-run all sorts of variations of trajectories to find the optimal path for the mission.
The only free software I've found so far is the Orbiter spaceship sim (link below), but nothing for generic use. There may be good algorithms out there somewhere just waiting to be coded...
