Holding back academically advanced kid - Advice?

Worried about my July-born son starting kindergarten? He's academically advanced (already reading at 4!) but I'm concerned about the long-term impact of being the youngest.

He's in Pre-K now and emotionally seems on par with his classmates, maybe just a smidge behind. His teacher isn't concerned about him starting in the fall. He's also always been tall for his age.

But I keep thinking about middle school, high school, and even college... Would more time for brain-development be better when it comes to making decisions and handling social pressure? Did anyone hold back a summer-born child, even if they were academically advanced? I don't want him to be bored and end up having problems in school because of that.

If you started your summer-born child on time, how did it impact them later on? Any regrets? I'm especially worried about the social and emotional impact of being the youngest in high school and college. Did anyone notice a difference in their child's maturity or confidence?

I've read some research (like Emily Oster's) but I'm really looking for personal experiences to help me with this tough decision.

Edit to add: school cut off for kindergarten in my town is that you have to be 5 years old on or before August 31.