Fear of alcohol/drugs. Anxiety help?

I'm gonna start by saying I feel like this is a weird thing, and it shouldn't be a big deal for me, but it really is.

I'm having issues with drugs and alcohol. I don't do them, the problem is that I get extremely anxious and have minor panic attacks just thinking about my friends drinking or smoking or anything. My stomach aches, my palms get sweaty, and sometimes I spiral a bit and cry. I haven't had any traumatic experience or anything, in fact, I haven't been around them at all. I've done a lot of thinking and I know that I'm really afraid of them because I perceive the dangers in the activities and overreact to them a lot. I'm afraid of the bad things that can happen when under the influence of those kinds of substances. It all stems from my fear of losing control. I have a history of addiction in my family and I don't want to lose control to that, but even more so I'm afraid of the simple loss of control the substances create in the moment. They make you lose control of yourself, to an extent, and I'm afraid of that. Where I'm at right now is trying to realize that I'm not in control of other people's actions. No matter what I do, people can still make mistakes (with or without alcohol/weed/etc.). I have to trust my friends, and my self if I ever choose to, to do it responsibly, and make good decisions. Letting go of my illusion of control is what I have to do to get over this fear.

Basically, I'm just looking for ways to help me trust people, get over my little faux-phobia, and avoid anxiety attacks. Help is much appreciated. Thanks you so much for reading this and responding.

I guess I wasn't super clear with that part. Yes, both of my parents struggled with addiction, namely alcoholism. But they've both been sober since before I was born. I've never been around a problem like that. I've never seen the negative repercussions. Sorry if that was unclear.