By setting
osd pg max concurrent snap trims = 0
and noscrub and nodeep-scrub, the cluster was able to finish replicating and we have then re-added all down osd's again.
But these 3 parameters/flags need to remain set to prevent "random" OSD's to go down.