Bismihi ta'ala
This is an interesting question, and it can be tackled from different angles.
In general, the ruling is that a female (mahram or non-mahram) cannot pray beside or in front of a male. This is not purely a matter of preventing men from unlawful glancing, because if that was the case, then a wife could pray in front of a husband.
Islam focuses on elevating the status of a women beyond her being sexualised by others, and even by herself. This can be seen not only in mandating hijab, but even in the obligatory or recommended (mustahab) acts in prayer, like how she prostrates, and how she stands back up.
This is all for the purpose of preserving her modesty, and also to eliminate any likelihood of a lustful gaze (haram or halal), during prayer in particular, for the purpose of gaining that absolute concentration in prayer.
Of course, we are assuming that this is just one of the reasons behind the ruling. It's similar to the story of Prophet Moses (a.s.) and the daughters of Prophet Shu'ayb (a.s.). However, the reality is that like many other rulings, we do not have the precise reason for it. There could certainly be more reasons.We are just assuming that it is a matter of convenience and a spiritual matter for concentration, but it could be more than that.
The main issue is in prayer we need to attain ḥudhūr al-qalb, which is presence of the heart.We need to be concentrating completely on our prayer, and remove anything that would distract us.
We understand that it is also the very natural structure of the female that Almighty God has created with her beauty and attraction that could contribute to a male (her mahram or non-mahram) being distracting and glancing at her.
Please look at all the other rulings for prayer, which can also hint to how important it is to avoid loss of concentration, like praying in front of an image, etc.
It could also be for convenience, so a woman does not feel embarrassed or exposed when she is bending down for ruku' or sujud.
And so on.
Furthermore, please refer to your Marja' taqleed on this ruling, because there are some jurists who consider it mustahab, and not wajib.
And Allah knows best.
