Making jets? Drill one and check it. I don’t think you will be oversize. Might be fussing over nothing and the set screw costs less than a dollar.
You might be surprised what a quality chuck will do for reducing runout. Figure out what taper your spindle is and get a ball bearing Jacob’s...