The crank case breather emits not only "oil fumes" but also any blow-by which gets past the rings, which is why an old or well-worn engine smells fumey when it runs. As already stated, it is usual to direct those fumes from the breather back to the engine air intake.
Tony wrote: but that would vent oil fumes and not diesel fumes... "diesel fumes"? If its the smell of diesel fuel then it won't be the engine. There would have to be a fuel leak somewhere between the tank filler cap and the injector pump.
Can Pol be a little more specific about the term "diesel fumes"? Is it the smell of diesel fuel? Is it the smell of exhaust? I would have thought that an exhaust leak or a fuel leak would have been diagnosed easily enough by someone as experienced as Pol. My guess is its probably just the smell of a tired engine with the odd oil leak, which gets smelly when hot, possibly exacerbated by fumes from the crank case - and perhaps a crew particularly sensitive to those sorts of smell.
It might sound obvious but you need to consider if the exhaust is drifting back into the boat too. Pleasure boats all seem to have an exhaust pipe which bubbles away at about the waterline down about the transom somewhere. Its probably the worst place for it - I don't know - all the longline boats in our local fleet had dry exhausts which vented upwards, overhead, for exactly that reason, and exhaust fumes were never a problem. The little pleasure boat I am living in now is no longer running, but the last trip I did with it and its tired old Bukh - it had a crude fixed shelter over the cockpit - and it dragged exhaust fumes up from the transom something awful.
I happen to like diesel engines very much, and the engine smell doesn't bother me at all - but for reasons which are not entirely rational, I plan eventually to go entirely away from fossil fuels this time, and I too will be following Paul's electric conversion with great interest.
I'd like to try and do without a generator though. I am rather more inspired by Kurt (Mehitabel) and learning to live within the limits of solar, in the same way as we learn to work within the limits of wind and tide and get our satisfaction from that. The prototype will be a Yamaha 9.9 and if that does the job, then step 2 will be (hopefully) to swap it out for an electric equivalent, to be used stringently. Lucky for me though, I'm retired and won't have to be under pressure of time or work requirements - I realize not everyone is so lucky.