Can a woman divorce her husband if he is abusive and refuses to divorce her? Is it incumbent on the Islamic court, in such a case, to divorce on behalf of the husband? Would the wife have to return the Mahr?
Islam does not allow injustice from any one on any one. Husband has no right to abuse his wife or deprive her from any of her rights as a wife. If the husband does not fear God and insists on doing wrong to his wife, she will then has the right to refer her case to the Islamic authority (Haakim al-Shariah who is the Marji' of Taqleed or his deputy) who orders the husband either to provide his wife with all her rights or to free her by divorce. If the husband refuses both options, Haakim al-Shariah then divorces her with out consent of her abusive husband and releases her from such harming husband.
Wife does not need to return her Mahr in such case.
There is a type of Talaq called Khul'a in which the wife does not like her husband and wants to be divorced from him and gives him an amount just to divorce her. That amount can be the Mahr or less or more. If the husband agrees to take that amount to divorce her, that Talaq is called Khul'a and it is a final Talaq with no return.
Wassalam.