My experience has been that surfing a wave and using a stern rudder to counter the tendency to broach works reasonably well. There is a lot of force at work when the boat starts to surf and it requires good bracing and timing to counter the tendencies to broach.
When water piles up on the beach, especially after large waves, it has to flow back down the beach into the surf. This creates a current (called an along shore current) which impacts incoming waves. The bow of the boat will tend to turn in the direction of this current. A stern rudder (or low brace) is generally sufficient to counter this. The boat will definitely broach if the low brace is on the wrong side of the boat.
On steep waves, the stern may rise up and plunge the bow of the boat into the ocean bottom. Generally, this means the boat will (generally) deflect to one side or the other since a sandy bottom has only slightly more give than granite. This too can be countered, but I have only succeeded in that once - the rest ended up with rolls. If the wave is really steep, the bow may stick into the bottom and pitch-pole the boat. I've seen someone actually recover from that, too, but I can't claim that degree of competence.
If you look at the videos at this link, you can see some pretty decent demonstrations of the tendency of a sea kayak to broach:
As the stern rises, and the bow sinks, the bow and stern may encounter different currents, and this can cause issues as well. This is more common in rock gardens where water must flow back through and around the rocks to return to sea. Generally, this is handled with a low brace, but in strong currents, a broach my happen PDQ. On rivers, this can happen when your boat is long enough to encounter eddys at both bow and stern. Good practice, but more than a bit disconcerting :).
In any case, a rudder isn't particularly useful in any situation where surfing may occur, so it is best to do what you can to keep it safe from being deployed. If you are paddling with someone, they can help ensure the rudder is stowed properly before you enter the surf.
1 Canoe/Kayak Trailer
|Table of Contents|