Logically, the ones that don't have such a nice winter appearance and don't have the worries about die-back (such as the herbaceous perennials that die to the ground) should be prettied up now.
So that should mean that this vitex should be cleaned up now, although it does get a significant amount of winter dieback which is not necessarily predictable, and pruning now can mean pruning again in the spring when new growth shows where the live parts and dead parts are.
Caryopteris has a nice winter appearance, but will need to be pruned back hard in the spring, and although I could just shear it down, I like to do the more time consuming task of pruning out the dead wood. So the debate continues. In all honesty, things will get cleaned up as time and weather permits, usually when I'm itching to do something in the garden, even if I "should" wait until spring.
There, don't you feel better now?