When I reset the cache to see the new patch... again the Harem was empty and I couldn't to anything about it... However great idea to create a background story for the game. I hope you can resolve the problem.
Can I suggest you to make a part of the game where you can customize and develop your character? I think at something similar to the profile page of Harem-Heroes, something like that, where you can equip or put all your charm creations.