Vocoder V and Logic Pro Latency

I just used the Vocoder V the first time in Logic Pro. Before I used the standalone app or Gig Performer for playing live - everything fine with GP or standalone.

In Logic Pro the setup for recording live is quite tricky, but OK.

This may be more a Logic topic, however, I wonder, if somebody over here experienced the same:

Every time I switch the vocoder SideChain input in the plugin to a physical inputting a reference audio track, ALL MIDI inputs are more or less unplayable. There’s a big latency, or better delay like playing a 1e or 1+ instead of a 1.

It’s nearly impossible to play/record a vocoder live. I think Logic tries to compensate something, but how to prevent/solve this issue?