Restrict accounts from your following list. Restricting limits their interactions with you without unfollowing them, and they aren't notified.
