what's your life like in general? stressed much?
what's your diet like? using any supps? any recreational drugs?
i saw the doc for pretty much the same thing a few months back. my testosterone came back at the bottom of the range but not low enough for the dr to want to do anything about it - he was more concerned with my cholesterol and convincing me i needed to stop eating keto.
my issues were solved by backing off my work hours, coming off my cut and eating more (still super low carb, just not in deficit), getting more exercise and supping more vitamins. i have a strong suspicion that the combination of low winter sunlight and lots of overtime left me pretty depleted of vitamin d, but that's really just guessing and i'm sure it was a combination of many factors.
much happier now. mood is up, energy is back, sex drive is way better but since i've gone back to long hours and the missus is pregnant i'm not getting too many opportunities to take advantage of it.
good luck getting to the bottom of it. hormones suck when they're working against you. don't be afraid to seek an endo if you're not satisfied with what your doctor is doing for you.