-- Last Updated: Feb-08-13 8:17 PM EST --
I think this is a great website for handheld GPS reviews as well as the following buyers guide:
The good news is that what you want is pretty basic functionality for a GPS unit.
The Garmin eTrex 20 is probably the lowest end you want to consider, the main reason is a GPS unit really is only as good as the maps you can load, and the eTrex 20 will allow you to load other maps then the base (unlike the cheaper eTrex 10). For kayaking I do lakes, rivers, streams and I use a bunch of free topo maps for my area but what is really useful is the ability to download the BirdsEye's Aerial Images (well worth the yearly cost). For kayaking you are frequently in areas without any ground cover and the sat photo image is helpful on the water.
I don't sea kayak or do large open water so no clue what people prefer on open water.
I have a Garmin Oregon 450 which is a bit more ($250 on Amazon) and I would reco you consider it as well. I had a hard time justifying the more expensive GPS units. I use my Oregon for kayaking, biking, hiking, Geo-caching, as well as just tracking general miles walked for fitness in my neighborhood.
One thing about handheld GPS is they are not great for people who expect them to work like a car GPS (you throw them on the dash and go). They are good for people willing to take the time to dig for maps they like and take the time needed to learn how to use them well. If you want something simple that you don't really need to mess with then you probably better off with a smart phone and map apps for them. Just don't count on it always having a 3G/4G signal when you need it or enough battery charge after being in GPS mode on the water for 6+ hours.