Wanting own house after being promised before marriage

Salaams Mufi Saheb. I’m married for a year and a revert. I’ve been through a long relationship with my husband for 12yrs before our families acceptd us. Before marriage my husband promised to move on our own. And after marriage I lived with my mum until my brother came to stay with her because of a problem. His home has 8 people living there and I can’t handle the noise and have anxiety problems as I went through post truamatic stress. My mum is now living in jhb and told us to live here until she comes back. She now wants to live in jhb permantley but my husband is refusing to live here. She removed all her prayer things before she left. I read n make dua daily here. He hit me 2days ago because I told him I don’t know when she is returning and he wanted me to go back home across the road. I refused and told him I’m happy staying here. He only wantd me there so our baby is with his family. He isn’t worried about my hapiness or the promises he made. He leaves us alone when we lived there to go smoke weed with his friends and hardly spent time with us. His family were aware of this but never intervened. I don’t say anything because I’m nt comfortable living there. He depends on his brother to support the house and I hear words from his sister inlaw. They don’t tell him anything. I am running a small tuckshop at my mums to earn an income and now I’m very happy living here and he isn’t. Am I wrong to want to live on our own? Please advise. Jazakallah