Yea, I'd always wondered about this too, so I did a search on the internet a few years back & came up with no scientific explanation for it. I'd heard some rumours way back in high school about building up levels, or resistance, or some such weirdness, but I'm pretty durned sure that 'aint what's up.
In my experience, I chalk it up to two things-- not knowing how to smoke (most cigarette smokers don't seem to have this problem so often), and being just plain too aware or paranoid... There seems to be some sort of person (this was our high school theory here) that just investigates their brain so darned hard that they pull themselves nearly out of the high... I dunno... that's what we came up with back in the day in our circles.

who knows.
And as for memory loss? Pssshaw! I remember my first time like it was this morning. I was 13 and living in Belize for a year. My friends' older sister had just finished curing some of the most beautiful buds I had ever seen, and she rolled a spliff I remember telling my friends was as big as my arm... The greatest, most cerebral sativa I have ever smoked... The jungle was alive with life, I swear I saw some birds that really weren't there, and sitting in her back yard I was convinced that I was making a jasmine tree grow by projecting love on it....lol. I was high the next day still.
Oh, if only I'd had the forethought to get my hands on some seeds, or the chance to go back and get some now....
Thanks for that. That's one of my favorite memories... and it's not too far off from the first time I ate mushrooms. Life IS good.