@Varnart
Girls can not grow in affection on their own. The level of affection can only be raised by giving gifts to the girls which will increase the amount of money they can create.
..in the entire time playing this game i did not see it happen any other way. If there are other ways to raise or even grow affection please fell free to correct me..^^