People could use it to track who unfollowed them and that could cause problems if someone wants to… so I think better ask jeffalo for that
it's already possible to keep track of who unfollows you if you want to (but just to be safe i might store the list of people who follow in a private mongo db)