Being based in Scotland and considering a boat in the near future, possibly for some long term cruising, the subject of insulation is one that is of much interest. My last boat, a 30 year old Hirondelle always smelt a bit “foosty”. Condensation had got behind the foam backed vinyl causing black mould. A problem I am sure anyone with a boat of similar vintage will be well aware of. A horrible job to get rid of which was one reason I did not feel like replacing with new. I accepted that condensation would occur as would some mould, but this could be kept at bay with good ventilation, as well as cleaned and repainted. For paint I used ‘bathroom paint’ a high vinyl content emulsion with a mould suppressant, it is amazingly durable. This was all ok for a boat mainly used as a day-sailor. For longer term use, insulation is essential.
Being and Architect I am well aware of the theory of avoiding condensation in buildings, though this is difficult to apply to boats. The ideal is to have a vapour barrier (normally thick polythene sheet) on the inside, between the insulation and the lining material. This prevents moisture from the air diffusing through the insulation, where it would cool down and condense. Of course it is never perfect, full of screw holes etc, so the moisture does get through. I a building this is dealt with by the outer layers of the construction being significantly more moisture permeable than the vapour barrier, and ventilating behind the outer cladding. Now this is more or less impossible in a boat, as the prime concern is keeping the water out, therefore the hull is (hopefully!) very good vapour barrier.
There is an alternative approach which is often used in flat roofs. These have the same problem as a boat, in that the outer layer is a good vapour barrier. The solution is to sandwich the insulation between a high specification vapour barrier inside and the outer covering. These are all adhesive bonded, so no nail holes. Whilst a small amount of moisture will still diffuse into the insulation, the amount can be calculated. During the winter it builds up and during the summer it dries out. As long as this is not more than will saturate and damage the insulation it is ok.
This approach is how you should consider boat insulation. If the boat is structural foam cored fibreglass, you are there already. Though a thicker than normal foam is desirable. If insulating behind lining, assume the lining is somewhat permeable (it is most likely to be removable). Fit polythene sheeting behind the lining and make sure all edges are well sealed, as are any screw holes. You can buy special tapes for this (duct tape is useless as it falls off after a short while). Any timber behind the polythene needs to be well sealed, the hull included, epoxy being best. Using treated timber for fixing battens is a good idea.
A good insulation is Extruded Expanded Polystyrene (Styrofoam) as is it tough and easy to work. There are higher grade insulation boards, though these are more delicate. The thickness will depend on where you intend to be, the size of boat and how much heat you are happy to pay for. I would go for about 50mm, though to bend to curves it will need to be thinner using two layers. A recent innovation is the multi-foil quilt, this gives the equivalent of about 200mm insulation in 30mm thickness. Being reflective it would work well in keeping heat out.