The easy way would be to switch primary bounce to brute force. Keep the light cache as secondary.
If you want it to render faster, I'd recommend this work flow.
1.1 IR primary and LC secondary for static objects.
1.2 bake IR map for static objects in 1 single map using "Add to current map" mode. Move your time line around so that the IR map would cover everything. Doesn't need to be every frame.
1.3 Once you have your IR map, switch to "from file" mode. Put your moving objects into a V-Ray object property. Have them cast shadows but don't receive/generate GI.
1.4 Render
2.1 Use brute force primary LC secondary for your moving objects.
2.2 This time put everything else in another V-ray object property. Make them a matte while generating and receive GI.
2.3 Render.
http://forums.cgsociety.org/archive/index.php/t-1133460.html
