i don’t think i can marry him. We called off the wedding .

Me and my fiancé got into a huge argument. We both have children with other people and none with each other. He has a son and I have a daughter. I wasn’t raised by my mother or father but my granddad which lead me to having abandonment issues. I feel like everyone is usually temporary in my life. He knows this but yet when he gets mad he shuts me out and starts retracting. This makes me very anxious and nervous about our future. Every time I express this concern to him he brushes it off. When I bring up how much my grandad means to me and how I need security he gets offended. He says I shouldn’t talk about my grandad when talking about him. He says I am going to cause him to hate my grandad. I’m very unsure why my grandad is such a trigger for him. They have never had an issue , my grandad has never said anything bad about him behind his back or to his face nor is my grandad in our business. My grandad is actually the one who is paying for our wedding. This raised a huge red flag for me and I don’t think I want this anymore. Am I over reacting . I don’t want him to be my dad… my dad is just the only people I know that has never walked out of my life.