Yup see your point.
Milling the tube/elbow joint is an option, if you used shims to arrive at a parallel situation at the wheel, then measured the shim pack to get the amount of metal removal required.
I have a surveyors level that sits on a tripod, to level it there are two wedged discs underneath it, you turn the discs to level the instrument regardless of the inclination of the mount.
So picture two such discs with long slotted holes for the bolts that hold the hub on, these could be rotated to adjust toe but retain camber.
Lordy my head hurts, I can hear the masses saying ' just buy another elbow ya tight git ' - Aha ! but where is the fun in that ?