Males don't flower like females, they flower like males. In other words they grow balls. If you wait to long to cull the males you could wind up with his Love Dust all over the place.
It is incredibly easy to tell the males from the females--they flower entirely different from each other. You will have plenty of time to cull the male(s) before they drop pollen. It is at least a couple of weeks between the first sign of balls and the time he will release pollen.