How much vacuum are you pulling at idle with the cams? Not knowing anything about the specs, the L&M by pass is for pretty low engine vacuum for fairly radical cams with a lot of overlap and making less than 12" hg of idle vaccum. But I can see the by pass closing much slower by virtue of less spring tension on the actuator. Probably wouldn't hurt to see if the by pass valve is moving freely too. Carbon deposits and burnt oil can gum up the mechanism and cause the valve to have some amount of resistance.