My boyfriend is 4 inches shorter than I am?

Well, my boyfriend of two years is 4 inches shorter than I am. When we are alone, I think absolutely nothing of it and think only of how much I love him. Unfortunately, when we are in public or around family, I feel like people are judging us. Actually, we both feel that way. The docor says he has stopped growing at only 5.4. I would never want to break up with him because he is the love of my life and Iwant to spend the rest of my life with him. I guess my question is, what can I do to stop feeling this way?