Focusing with Voigtländer X-mount pancake lenses

Lenses in question

https://preview.redd.it/jso7n5bwctzd1.jpg?width=6240&format=pjpg&auto=webp&s=b45325f10609978aa55fd09d06f9132053e96670

Cat toy

I have both 18mm and 27mm Voigtländers. I like the size and quality of the lenses and the pictures that come straight out of the camera.

The only problem is that I find getting the focus right hard when I am not shooting at f2 or f2.8, especially when focusing at distances beyond 2-3 meters.

I have both focus peak highlight and focus check turned on. But at apertures f4 and smaller, it's really hard to nail the focus. I sometimes focus at f2 and then change the aperture, but usually don't have time for that. What do others do? Is this a typical experience with these lenses? Does anyone have tips?