Large double-logarithmic corrections are induced by soft gluon emissions near threshold in the semi-inclusive $e^+e^-$ annihilation (SIA) distributions, and must be resummed to all-orders in perturbation theory for reliable theoretical predictions. Building on strategy developed for threshold resummation for DIS structure function in momentum space using soft-collinear effective theory (SCET), we present the explicit formalism for SIA cross section. We then perform the resummation directly in momentum space for $\gamma^* \to q \bar q$, $H \to gg$ and $H \to b\bar b$ to N$^4$LL accuracy and demonstrate good convergence. We anticipate that these results will benefit the extraction of the light-quark, the heavy-quark as well as the gluon fragmentation functions.