Working with what stays
Use the repeated windows as the contrast test
The Colonial example has dark shutters repeated across a pale yellow facade. Compare the body with those dark areas intact before deciding whether the trim should be a brighter white or a softer cream.
Keep the yellow restrained at house scale
A small chip can understate the visual effect of a warm body across two stories. Compare the entire front, including the shaded entry, before narrowing to one yellow.