Wind shear may be the explanation to why a square sail seems to be more effective in my experience. As Paul said, there is more sail area higher up in faster wind, and it may be that simple.
The Chinese Treasure ships from the 1400s are almost always shown with square sails. Images of the Keying and the Ning Po, originally named the Kin Tai Foong, seem to show square sails as well. There was a replica built of the Keying called the Keying 2 but fan top style sails were used.
Pictures of these boats is why I tried a rectangular sail.
These ships are also sometimes depicted with their sails centered on the mast, which must have made them very stable downwind. Like Lores that Graeme posted. I bet Lores was very fast and stable downwind, but without the ability to shift the sail off center on the mast, I doubt Lores made much headway against the wind.
The old Chinese boats must have had some technique for moving the center point of their sails on their mast. It could be that whatever technique they used to do so also enabled better windward capabilities, but that is pure theory. Such an ability in an already comparatively complex rig design is a head scratcher to be sure. I have seen a junk rig with the ability to move the sail slightly, but it was not perfectly centered and did it seem easy to do so.
On a side note, since my sails were fixed in their position on the mast, while sailing downwind, if I let the sail out too far, it started pushing the boat to windward and would almost dunk me in the water! I guess that's called a death roll. If the sail was able to be centered on the mast, that may not be an issue at all.
It is interesting to ponder what makes a true junk sail. From what I have read about the Keying when it visited England, reporters claimed the sails were made out of bamboo and matting! It may be possible whatever technology these old boats used is not feasible to recreate. Perhaps whatever plants they used to weave into sail cloth, or the way they wove the fibers, had some inherent aerodynamic quality.
This is an awesome website, and I have a feeling there are some very cool discoveries on the horizon.
-Colin Clayton