Is the ring too small?

Recently got a beautiful ering upgrade. I had a size 5 previously and it was too big, nearly falling off when cold and spun to the palm immediately after putting it on so I ended up getting a size 4 with the new one.
Here's the dilemma, I have big knuckles and skinny fingers. The 4 is a struggle to get passed my knuckle in normal circumstances, it takes a bit of finessing. When I'm swollen it's nearly impossible so I just wait it out until my fingers go down a bit. It fits fine once it's on though, still a little loose and still spins a bit it's just my annoyingly large knuckles that are the issue.
I'm thinking of adding a band or two just to take up some more room to try to minimize spinning, I'm not sure how that would work out and I'm also not sure if my ering is too small or not since it is still a little loose once it's on. Any other people with big knuckles who can give me some advice?