In your family, your grandpa being cared might mean that he important to your family/respected by your family.
By your perspective, I feel he didn't have any way to convey how he felt about you....Maybe cause you didn't speak Chinese? idk
From my observations, grandparents love their grandkids...