The bottom couple of feet of the mast is probably out of sight, out of the weather and not in the way of anything, yet easily inspected at any time. And a little bit of bulk and/or a couple of kg of extra weight down there is nothing. If the top 10.8 m is OK then there must be a dozen ways of beefing up the bottom 0.2m. Apart from the other normal considerations, the compromised lower 10” of mast needs to be strengthened so that it cannot crush or stretch in circumference. You then need to consider how it will be stepped.
In writing: “Make the step as a socket, and the situation should be better, but I still wouldn't trust it” I do wonder if David T is being a little too conservative. David, would there be anything wrong with inserting a nicely fitted 0.4m (say) internal sleeve, or wooden plug, and then (as you suggest) housing the heel in an outer sleeve or socket? This outer sleeve could possibly even be a metal clamp or series of clamps strong enough to resist stretching or opening up around the circumference. Alternatively, a larger diameter piece of aluminium tube for an external sleeve can be packed to fit – or 3mm aluminium sheet folded into any sectional shape will do if fastened to itself and packed to fit. Whatever, the internal plug or sleeve must resist crushing, the outer sleeve or housing must resist stretching. The mast step itself, which could be part of the solution, must meet all the other usual requirements. I think that is still being careful and conservative.
Paul, I do feel strongly that if the sail plan is already decided, you should not compromise by shortening the mast. You could, as David suggests, change the sail plan if you think you can’t make the foot of the mast strong enough – but do be generous with your allowance for halyard span at the top of the mast, and paired-downhaul spans between the boom and the deck. Good luck with your split rig, you will like it.
PS here's another thought: make the mast slightly more than 11m so that the bottom 6" is below the hatch cut-out and comprises the full circumference. Then just sleeve or plug the bottom of the mast up to a point, say, 6" or more above the hatch cut-out (the more the better).