Elopement or intimate ceremony w/ fam?

FH and I want to elope at our favorite place, right on the river in the mountains a couple hours from where we live. We kept going back and forth on if we want it to be just the 2 of us (plus officiant and photographer) or if we want to include our immediate family (parents, siblings + our 2 year old daughter) and like 5-6 of our very closest friends ONLY. We were pretty much sold on doing it with just the 2 of us, but when I mentioned this idea to my mom (cautiously) she said she thinks my dad would be pretty upset if he couldn’t walk me down the isle. I honestly didn’t even think of this and now I definitely want him to walk me. So naturally we would switch to idea number 2 with immediate family and close friends only. Well, I told my future MIL about this potential plan and she mentions that we should invite my FH’s grandpa because he lives close by to the river we are eloping at. I said no, immediate family only includes parents, siblings and our daughter. Plus, if his grandpa is invited, we’d have to invite my grandpa to avoid hurt feelings and so on. This is one of the MAIN reasons we wanted to elope because my family is huge and someone is bound to get offended for not getting invited, PLUS the less people the better in our opinion. We want this as intimate and small as possible. My future MIL acted really offended that I said no to grandpa... I still stand by what I said but is it too harsh? I don’t think so but I need reassurance.