How and why is it considered respectful in Islam not to shake hands with the opposite gender, especially in formal settings when it seems to be a reasonable social expectation with no aspect of lust involved?
as salam alaikum
from a psychological perspective, even the mere presence of a person of the opposite gender have a particular influence that mold behavioral tendencies and attitudes that not necessarily relate to lust. That is also why many Islamic places, and traditional societies in general, always had a specific code to limit promiscuity and social contact between genders.
There is no problem in dealing with the opposite gender in the prescribed shar'i way and in case of necessity as long as all Islamic etiquettes are observed but one should try as much as possible to move forward what Allah has commanded in all circumstances.
With prayer for your success.