Glory, as Chris says, was heavily modified. Sir Henry bought a bare hull and went from there. The boat was massively constructed. The ballast figure quoted was confirmed by Sir Henry in emails to me while I was writing the HOF article. He had a unique vision of what he wanted to achieve in his voyaging, and one of those things was to operate a mobile radio station, which attracted huge interest from amateur radio enthusiasts during his voyages. That alone required significant power generation. On top of that, Sir Henry has a preference for the motorsailing concept and carried more than 500 litres of diesel fuel. It is not surprising that Glory's overall displacement was well over 4 tons.
Motorsailing allowed him maintain high averages but there were still numerous passages where he did not have the fuel range to motor all the way. With a steady following wind, Glory was quick enough under sail, managing to average a good 4 knots at times. For much of his Pacific crossing, he had trouble with the engine and was forced to sail more than usual.
One could build a lighter, faster boat - I think of John Guzzwell and Trekka - but to carry Sir Henry's cargo, it would need to be much longer on the waterline. And John Guzzwell himself, on being reunited with Trekka in Honolulu in the early 70s, said she seemed awfully small. For a circumnavigation, Trekka is perhaps a young man's boat. Jester and Mingming were sailed by old men but they made short voyages and their skippers lived most of their lives ashore. A boat must suit her skipper's purpose and personality. Glory is not for everybody - I prefer a bit more sailing performance myself - but she suited Sir Henry perfectly and allowed him to live aboard and voyage extensively well into his 80s.