There are a lot of things to consider in what you've asked.
The type of mold.
The amount of spores ingested.
The overall health of the person who has ingested them.
The strength of their immune system.
But yes, if the mold is of a type that can harm people if ingested, it can harm you if your body isn't able to fight it off.
When smoked, the mold may not even survive the heat or if so, it may not survive it in numbers that would do more than make you feel down a little.
Or, if its a type of mold that is deadly, one spore may spawn enough others to kill you.
There is no simple answer to what you've asked.
The best bet is to prevent mold from being in your weed. Then you don't have to worry about it.