That matrimony as such is not sinful the Fathers are agreed, the odd eccentric apart, yet the concession turns out to mean disappointingly little, to judge by the pronouncements of some of the most venerated authorities among them. Not only is the use of marriage sinful during pregnancy (St. Jerome, Comm. in Eph. iii) or lactation (St. Gregory in Bede's Hist. Eccl. Gent. Angl., 1.27), but the very institution of marriage is due simply to the need for avoiding fornication (St. John Chrysostom, Hom. on I Cor. 7.2). A greater evil is escaped by tolerating a lesser: it is bad to use marriage (for St. Paul says it is good not to touch a woman [1 Cor 7:1]), but worse to fornicate (St. Jerome, Adv. Jovin. 1.7). St. John was not martyred, because he was unmarried, but St. Peter, although he left his wife when he followed Christ, still wiped out the dirtiness of his marriage by shedding his blood (ibid., 1.26). For St. Augustine ... nothing casts down the manly spirit more than love making (I Solil., 10), though the shame attached to it is fallen man's penalty, not itself a sin (De Nupt. et Conc. I, 5), Augustine never, to my knowledge, claims that sexual activity is sinful as such. At the same time, he limits its sinlessness to when the action is performed for the procreation of children or to pay the marriage debt, not when it is done for mutual enjoyment (De Bono Conj. 6). A papal endorsement of this is given by St. Gregory the Great in his letter to St. Augustine of Canterbury .... There is no fault in the pains of childbirth, he says, but there is in flesh[l]y pleasure. After intercourse, a man may not enter a church without penance and washing, for till the fire of concupiscence has died down he is unworthy through the wickedness of his evil will. Marriage is not unlawful but its use is impossible without blameworthy desire.
(G. Egner [pseudonym of P. J. Fitzpatrick], Birth Regulation and Catholic Belief: A Study in Problems and Possibilities [London: 1966], 127)
And:
The charge that the Fathers we have quoted said sex was sinful is, as it stands, too vague to be justified. But the cheerful denial commonly made to the charge is extremely misleading. True enough, the Fathers did attack heretics who taught that marriage was sinful; true also, they did not hold the act of marriage to be sinful as such. But they subordinated its use so severely to procreation that for fallen man to use his sexual faculties without sin turns out to be a difficult exercise in detachment rather than a gift of God used to love another of God's creatures to whom we have been united by him... (ibid., 129-30).
In the context of the book in which this analysis appears, Fitzpatrick is trying to show the difference between the present-day Catholic position on birth control and the older Catholic tradition. The modern position is much more liberal in that it forbids only the use of contraception, whereas the older position forbade having sex for any reason other than to have children (you could, however, acquiesce to your spouse's sinful request for sex without sinning yourself), for which reason contraception was obviously excluded. Fitzpatrick actually thinks that the older, ultra-conservative position, though repugnant, is more logically consistent, for he cannot see why, as a matter of logic, if it's okay to have sex with your spouse for fun or to show love, etc., as the Church says, it's not okay to use contraception. The Fathers would not approve the use of Natural Family Planning, as the Church today does. He thus seriously questions the validity of relying on the Fathers at all in trying to argue against contraception when their problem with contraception was a direct corollary of a view on sex no longer accepted within mainstream Catholicism: "If present-day Catholic thought gives more honourable a place to married sexuality than did the Fathers, are we entitled to doubt the value of appealing to them on questions about the purpose of marriage and its use?" He is being overly reserved, for obviously he means that the answer to this question is Yes. Of course, there are still Catholic traditionalists today who do espouse the older view, but they are marginalized.
As for my opinion, I will say that, at the very least, one who thinks (in the face of much evidence to the contrary!) that the Fathers having something edifying to say to Christians today about sex will have to approach them with an already-established hermeneutic and theory of sex in hand, by which he or she will be able to find any diamonds in the rough of their harsh and seemingly inhumane words. Otherwise you're just going to be either appalled or become a traditionalist, depending on your personal bias.