Actually this seemed to work only for post types.
But when editing my taxonomy, the slug still defaults to the Singular Name...
To be clear the issue is the default field value for the slug when editing a Taxonomy in Meta Box > Taxonomies
It's not actually changed in the database unless I accidentally save the default value, which is incorrect (I'm using custom taxonomy slug)