No doubt there is some truth in that. It cuts both ways though. Years of being turned down for sex by a wife who supposedly loves him doesn't do much for a guy's ego. Makes him gun shy. Once you're turned down often enough, you simply stop trying, how much disappointment can one person stand.
One fellow I knew, who unfortunately passed away too early in life, was married to someone who said she was asexual. He would tell me that in bed she would say things like, "Can you hurry up and get this over with" or "I don't know why you bother caressing my breasts, I feel nothing there." Not a fun evening. In his case, he managed to find a girlfriend or two who worshipped him, with whom he had terrific sex. But you can see how someone who wasn't as talkative as him, a little more shy, not good at small talk say, who had very little chance of meeting someone socially, would end having a miserable life, sex-wise anyway. I was among that number, and fun meetings with really pleasant sex workers did wonders for my morale back in the day. At one point in my life, the nicest people I met were sex workers.
For that and other reasons, I have never been able to understand the desire to make sex work illegal. How is it any of the government's business who we have sex with or why? It's utterly insane.