Necessity of ghusal

I am now married masha – allah for 9 years and have recently encountered a very embarrassing problem. I find that after doing the ghusl, by the next prayer time, I find a dried up discharge on my underwear, ( I see it as sperm as discharge does not dry up) My question is – do I need to perform the ghusl again. Please advice as I sometimes need to perform the ghusl up to 3 times now(fajr, zohr and then again for asr)
shukhran jazeelan. Wassalaam