I actually have thought of an invention for this, maybe it's already patented but who knows? Seems like someone would have thought of this besides me.
You basically just put the left side inside your pants and push it through (might have to poke a hole) then the button with a built on washer screws on and tightens up secure. What do you think?