Why is declining birth rate bad?

I've seen a lot of posts recently about declining birth rate and the rate of replacement and I still don't understand why it's a bad thing. Civilizations have existed with much smaller populations before now. Why can't we just downsize back to how things were before the boomers?