You should be able to keep your perks experience when switching.

it would just feel right... since we have to adapt but losing perks feels punishing..