Assuming the car does not idle at an abnormally high rpm, then whether or not the vacuum advance is connected or disconnected should not make a difference (or not much of a difference) in how the engine idles unless it was hooked to a manifold (full) vacuum source or you forgot to plug the vacuum line in which case there was a vacuum leak.
If the initial timing changes when the vacuum advance is disconnected and plugged you need to figure out why.
You said you changed the balancer. Did you check TDC on the balancer with the #1 piston at TDC on the compression stroke? If not your timing may not be what you think it is.
You said you changed the cam. Is there any chance the cam timing could be off meaning the cam was not installed "straight up"?
Do you have an adjustable valve train? If so how was the lifter pre-load set?
Finally, with a new cam there is always the possibility that the lobes and/or lifters were damaged during break in.