I knew logically that I shouldn't be upset about it, after all only my yetzer hara benefits from me being upset. But how to deal with these feelings in a positive way rather than ignoring them?
It ocurred to me that the Rebbe from Lubavitch would write long detailed correspondence with those who were further from him, whereas with his close hasidim a few words or a sentence would often suffice. When I looked from that perspective, that my father needs to spend the most time with those further from him, and because of my closeness I need less of his time, it allowed me to relax and appreciate our relationship.
This kind of thought process wouldn't necessarily bring everyone the comfort it brings me, but for me when I understand life through the lens of Torah it makes everything easier.