There is another way how affection points could be gathered. Now, for example if you have got already a guy from the club champion, all you get there is one louzy affection item. That really doesn't motivate people to keep fighting champions. So there coumd be at least the same approach as with shards, that mire you invest there, the more affection items you will get.
But i do not agree that there are girls/guys you are not actively using. Like look at the champion fights again. If you have like 7 developed charactersm but like 600 weak ones, it's very hard to create a decent team for champion fights. Besides with all the new blessings you never know when even usually weaker character may become a valuable fighter, so to evolve our harems evenly as much as possible is quite useful.