I haven't started on the modular yet, but I was thinking along these same lines. It's expensive to get into, but the minibrute seems like it will be a very useful stepping stone.
To get into modular from the ground, you would probably need to spend around $1500 minimum on a case, power supply, midi-cv converter, VCO, VCF, VCA, EG and some mults and cables. I'm sure there's something I'm forgetting as well. It'll add up.
But since I have a minibrute, I'm thinking of starting with a simple DIY EG, case, and power supply. Haven't decided what to go with yet, but this'll be a very inexpensive start which (unless I'm missing something here?) I can use as a pitch envelope - something that the minibrute is lacking.