Previous studies have demonstrated that two representative passive control devices, including inertial mass dampers (IMDs) and negative stiffness dampers (NSDs), exhibit superior control performance in single-mode vibration control of stay cables. However, observations in recent years have increasingly reported rain–wind-induced multi-mode vibrations of stay cables on actual bridges. Therefore, it is of considerable significance to investigate the control effectiveness of the two representative passive dampers in mitigating multi-mode cable vibrations. For this reason, this study presents a comparative study of the IMD and NSD for the multi-mode vibration control of stay cables. The mechanical models of typical IMDs and NSDs are first introduced, followed by the numerical modeling of the two cable-damper systems using the finite difference method. Subsequently, the effectiveness of three multi-mode optimization strategies is comprehensively assessed, and the most effective strategy is selected for the optimal design of the IMD and NSD. Finally, the effectiveness of the control of the IMD and NSD in suppressing harmonic, white noise and wind-induced multi-mode vibrations of a 493.72 (m) long ultra-long cable is systematically evaluated. The numerical results indicate that the NSD significantly improves the cable damping ratios for multiple vibration modes as its negative stiffness coefficient increases, while IMD performs well only within a small inertia coefficient. Moreover, the NSD outperforms the IMD in suppressing multi-mode cable vibrations induced by harmonic, white noise and wind excitations.