Looking thorugh Bill Belcher's book, I see that he mentions a 3:1 linkage ratio at one point, but in his design for a complete pendulum gear, he has a linkage ratio of 2:1 which is better. I use a linkage ratio of 1.5:1 on my gear. That is, the drum at the vane has a radius of 80mm and the tiller on the servos is 120mm long. To use such a small ratio, the vane has to be bigger, but the steering is much more positive. Also, there needs to be plenty of negative feedback built in to the system, to avoid oversteer.
With a pendulum servo gear, you bias the pendulum when there is a requirement for a lot of weather helm, by shortening the line on the weather side of the tiller and slackening the lee side line; then the blade stays more vertical in the water, and the vane stays more upright. With a trim tab gear, you can't do that. Also, with a trim tab, the tab is working against the rudder, not with it, so more rudder angle is needed, so more tab angle is needed, so more vane angle is needed. This results in poor course keeping. These two things are why I ditched the trim tab on Tystie and went to the more complex servo pendulum system.
I'm wondering whether your servo blade or trim tab (which is it?) is big enough, if the vane lies flat but there is not enough steering force being generated.