This is horrible. Please provide a way to get the terms added/removed the WordPress way in sync with the custom field. Just stumbled over this problem after using QuickEdit feature in Post list to update a lot of CPT Posts together with the client and now my data are inconsistent.