Mangrove ecosystems are vital for carbon sequestration and coastal protection, yet accurate estimation of soil organic carbon (SOC) using remote sensing remains challenging due to spectral interference caused by dynamic vegetation cover. This study presents a novel framework integrating fractional-order derivative (FOD) techniques with machine learning algorithms for SOC estimation in mangrove wetlands. A total of 201 soil samples were collected from five mangrove wetlands in southern China. FOD was applied to both soil and leaf hyperspectral reflectance to amplify subtle spectral variations typically overlooked by conventional approaches. SOC-sensitive wavelengths were identified using the SHAP-XGBoost (Shapley Additive Explanations-Extreme Gradient Boosting) method. A total of 363 modeling strategies were constructed using Random Forest, XGBoost, and CatBoost (Categorical Boosting) algorithms across 11 vegetation cover levels (0–100 %) and 11 fractional orders (0–2 at 0.2 intervals). Results indicate that fractional orders between 0.8 and 1.4 consistently yielded superior performance. The CatBoost model under 10 % vegetation cover and a fractional order of 1.2 achieved the highest accuracy (R2 = 0.730, RMSE = 0.858 %). Incorporating key soil and terrain variables (e.g. soil iron, clay content, pH, salinity, redox potential, and elevation) into the spectra-based SOC estimation model significantly enhanced prediction accuracy, highlighting the complementary roles of spectral signals, soil characteristics, and topographic features in SOC modeling. This framework holds the potential for advancing blue carbon accounting and supporting sustainable mangrove conservation and management under changing environmental conditions.