power booster is the prime suspect..
check first the vacuum check valve.
it plugs into the power booster via the large vacuum hose.
it prevents vacuum from flowing bi-directionally.
if that's bad, it's like $5 from an auto parts store, via Motormite
Otherwise, power booster.. remove the master cylinder, if you're careful, you won't have to remove the lines and you won't have to rebleed.
the 4 bolts holding the booster on come off, remove the booster and you're done.