It is permissible to wear it except in prayers. It is considered to be Najis so one should purify whatever it touches with dampness, like if one touches it with wet hands.
Since you did not know the correct direction of the Qibla within the mentioned deviation, the previous prayers are correct but the prayer should be, from now on, towards the correct direction of the Qibla, since you have realized that and paid attention to it. The prayer is void if the earlier Qibla is taken, as it is deliberate disobedience.
