I can't finish the game because Honeyed Words does not progress

So stupid sexy Sir Hans is just standing around in his camp after I gave him the Musk of Infinite Allure. I can talk to him when it is not the evening, but when evening comes, the option to talk disappears. The Wiki says he is supposed to travel from the camp at the edge of the map to Rattay, but this never happened for me. There he is prancing around in the camp with his shiny armor. Is my (first) playthrough ruined?

Praise to anyone who can help with this issue.

Edit: PS5