If I were you I wouldn't smoke anymore until I found out what was causing the problem. I say this because it could be what you say is true due to the fact there is some underlying pre-existing psychological condition that drugs bring out, or you're imagining it. On the other hand, maybe the doctor's right and you should get a blood test. I tend to think that it's your imagination; I don't think that it's an underlying psychological condition. I believe it's possible for some people with underlying, latent mental issues (I'm not saying this is you, so don't freak.) to develop mental illness from smoking too much weed, but it seems highly unlikely that this would be the case with you, because you don't seem like someone who has smoked it enough. There are people who don't develop mental illness from smoking too much pot, but might experience permanent brain damage instead. I believe this can occur with some people who smoke way too much too early in life. But then again, you didn't smoke way too much too early in life.
The more I think about it, the more I think it's your imagination. Something like that can happen to some people when using strong psychedelic drugs, even after only one time, but that's not the case with marijuana.
If it bothers you, then why not give up smoking and forget about?