If you can't be comfortable around him, it won't work. You don't want to come home from being anxious all day only to deal with it at home again. If you think you can change this, it's worth a shot, but I have no way of knowing that. However, he will notice your being uncomfortable about him, and probably think it's something you don't like about him, which will lead to resentment. It'll just be a slippery slope from there.

You say you don't want to fall into depression if you break it off, but you can fall into it just as easily, if not moreso, if you are in a unhappy relationship, which is sounds like you are.

To me it seems like you are just trying to convince yourself that you should stay with him because you love him. Now I'm not saying you don't, but you are giving serious thought into leaving him. A person in a healthy relationship doesn't do that. I'd break it off, at least temporarily, and see if the situation changes with some time/space.