When you remove someone from the Team, you are just "deactivating" them on the team list. Nothing else changes.
Did they leave their firm?
If the person is to be removed because they have left their firm, deactivate them in your organization's master contact list. This will deactivate them on all Project Teams and they won't be offered as a possible recipient.
Or did they just leave the project?
If the person still works at their firm but is just not working on this particular project, deactivate them in the Project Team List.