This paper presents ground calibration and orthogonality correction methods for the tri-axis fluxgate magnetometer (FGM), named as adaptive in-phase magnetometer (AIMAG), aboard the CAS500-3 satellite. The orthogonality errors of the FGM among the axes can lead to significant inaccuracies in magnetic field measurements. In this study, we employed Helmholtz coils and an autocollimator to apply controlled magnetic fields and adjust the magnetometer’s alignment. By deriving the correction matrix, we could transfer the sensor axes to the ideal orthogonal coordinate system. We validated the correction method by analyzing the sensor’s output under various magnetic field conditions. This correction method is expected to enhance the in-flight magnetic field measurements of the CAS500-3 satellite.