Stick with the stock manifold. Magnaflow has terrible customer service, and they don't have a CLUE about what they sell for the XLR. It's hard to get the manifold off, but not impossible. The manifolds I got from them came with bolts that wouldn't even go through the holes in the manifold, let alone fit the head bolt holes. All they have told me is to return them (after spending months with them telling me it fits, and if it doesn't something is wrong with my car) or pay MORE money to have them modified to fit (which they pretty much suggested from the start, which tells me they 100% know they don't fit some XLRs).
Some people have had just Cats welded to the stock manifold. Some have had the CATS gutted and the ECM flashed to ignore the back sensors (illegal for road use). Some just have the manifold replaced without taking the engine out, which is hard, but not impossible (at least on the NA engine).The passenger side is hard to get off, and I bet that's the one giving you problems?
If it's just the passenger side, get the stock manifold.
I have no idea how hard this is on a V. Might be harder. Can't be easier.