You've just got to have the right gear. One time, on a trip to Bay of Quinte, I fished outside all day in balmy -20 temps. With the clothing I had on, I honestly never got cold.