Restrict accounts from your followers list. Restricting limits their interactions with you without unfollowing or removing them, and they aren't notified.
