Just to update anyone who might be interested. Just had a conversation with the Migrationsamt and told them my situation and asked them about the 3 month rule. They basically said it's no issue at all and were almost surprised that I was worried about this. They said that they typically send out a letter to the person's registered address asking if they still live there before canceling the permit.

I was always under the impression that the border agents are the ones that would notice the period of absence and stop you from entering.