Being a descendant of a person or a family depends on the definition of descendant. In general you are a descendant of all your grandfathers and grand mothers. But if you ask about being a Sayyed, then the definition of a Sayyed is the person who was born from a Sayyed father. So, your grandfather is related to the Prophet's family from his mother, but since his father was not a Sayyed, he is not a Sayyed. This is the verdict of most of our great scholars.
Wassalam.