Making wudhu with trouser that might be napaak

the hand washing area is outside the toilet in open and I obviously have to get dressed before coming out to wash my hands. now, when I wash myself after passing the stool with my left hand ill have to pick up my trouser and atleast bring it up to button it up or close the zipper and yes sometimes my left hand does touches the trouser. What do I do in such a scenario? Can I still just make wudhu and pray or is it a must to change my trouser?