Early in my knitting career I became quite taken with the fisherman's sweater -- gansey, guernsey, jersey, knit-frock, call it what you will. I think of it as a solid dark colored textured pullover, with knit-purl patterns and small cables.
Probably the first book published on this topic was Gladys Thompson's Patterns for guernseys & jerseys, later revised to add Aran patterns as well. This is also the first book I bought on the subject. It's a great book, a wonderful book, and it probably has all the information you need, but it's not much of a hand-holder. I knit my first gansey using the sport weight version of Sugar & Cream and Seahouses Pattern I / Mrs. Laidlaw's pattern (p. 68 in the inexpensive Dover edition, which has held up well for all these years). Unfortunately this pattern, like most in the book, is not charted and I did not realize that it needed to be centered -- you can see here that it is off center:
The book is a classic and inexpensive & really a good addition to any knitter's library but not the first I would buy on the subject.
I think the next book I bought on the subject was Rae Compton's The Complete book of traditional guernsey and jersey knitting (1985). I love this book; it's the one I turn to the most on this subject. It has lots of contemporary photos of fisherman wearing ganseys, and charts of the designs on those sweaters. It has a great chapter on designing your own gansey that walks you through the concepts and the math. I used this to design my second gansey, a sweater for my father. The only thing missing in this book, IMO, is a statement to the effect that underarm gussets were used to allow someone wearing a very tightly fitting sweater to move his arms and that therefore they would be unnecessary in a sweater with plenty of ease. It took me years to knit that sweater, and when my dad put it on he grabbed a big handful of excess fabric pooching out at the armpit and said, "What's this for?" He wears it anyways, and I've stopped cringing, but it took a while.
Two other tightly focused books came my way at about the same time: Mary Wright's Cornish guernseys and knit-frocks (1979) and Henriette van der Klift-Tellegen's Knitting from the Netherlands : traditional Dutch fishermen's sweaters (1985) (a translation from the Dutch). They are both interesting but perhaps not essential books, certainly not the place to start. Both have a wealth of old photos, many of which I've seen nowhere else. The Dutch fisherman in particular are very cute and goofy; their sweaters are a little more elaborate than those from the western side of the Channel.
I've run across a number of survey books that cover or attempt to cover all types of traditional knitting from Great Britain & associated islands. Most of these were published in England; many are neither thorough or inspiring. But some are quite nice.
Priscilla Gibson-Roberts' Knitting in the old way (1985) introduced me to ganseys, as to so many other traditional sweaters. The section is brief (7 pp., plus another 3 on Dutch fisherman's sweaters) but enough to give you the basic idea.
Michael Pearson is (was?) apparently a famous British knitter of whom I had never heard before I ran across his book Traditional knitting (1984) at a library sale. It's full of fanciful tales of drowned sailors identified by their sweaters and the Spanish Armada as the origin of Fair Isle patterning and so forth, but it also has some very nice patterns, interviews with aged knitters, and lots of good photos. I tend to think of it as a Fair Isle book but actually half the book is fisher ganseys (100 pp.) with the other half evenly split between Aran & Fair Isle. Some of the photos are the familiar Polperro ones which appear in Compton's & Wright's books as well, but others are unique to this book.
Gwyn Morgan's Traditional knitting : patterns of Ireland, Scotland, and England(1981) is less substantial. She gives a small amount of historical information (3 pp.), followed by patterns for a dozen traditional-style sweaters. Each is named after a town. I think I keep this one partly because I can't stand to get rid of a book and partly because it has several children's ganseys, in several different weights of yarn. I wouldn't search this book out but if it turns up you could do worse than to pick it up.
Of course the master of the "inspired by" sweater is Alice Starmore, and I love her Fisherman's Sweaters (1993) even though I've never knit anything from the book and probably never will. Most of the sweaters are lovely, most of the photo styling is lovely -- it's lots of fun to daydream over.
Also, of course, there's this photo, which is a very useful reminder that some designs are just not flattering. If this sweater can make this willowy model look hippy and frumpy, imagine the effect it would have on me.
I can't end this essay without mentioning a book that I've never owned, and never even read all the way through: Beth Brown-Reinsel's Knitting ganseys. By the time it was published I thought I knew all I needed to know about knitting ganseys, so I didn't buy it. But I've heard wonderful things about it over the years.
If you are just building a knitting library on this subject, I'd start either with Rae Compton's book or with Beth Brown-Reinsel's. (Although if you don't have PGR's Knitting in the old way get that first, it's a must for anyone who is interested in knitting traditional sweaters.) And if you know of others I haven't mentioned, I'd love to hear about them.