Abstract Background The potential role of immune cells and the precise contributions of genes from various immune cells in the development of colorectal cancer (CRC) remains unclear. We aimed to investigate the potential causal relationships between specific immune cell genes and the development of CRC by performing Mendelian randomization (MR) analysis. Methods The cis-expression quantitative trait loci (eQTLs) data for type-specific immune cells were used to perform MR analysis. Outcomes data were obtained from the latest Genome Wide Association Studies (GWAS), comprising 78,473 European ancestry CRC cases and 107,143 controls. A few sensitivity analyses, encompassing pleiotropy and colocalization analyses, along with the application of False Discovery Rate (FDR) correction, were carried out to mitigate potential biases. Results A total of 395 genes were found to be associated with CRC. After FDR correction, a total of 47 genes across 14 immune cell types were identified. Notably, the FHL3 gene showed the strongest association with CRC in multiple immune cell types, including CD4 + naïve and central memory T cells (CD4NC), CD4 + T cells with an effector memory or central memory phenotype (CD4ET), CD8 + naïve and central memory T cells (CD8NC), CD8 + T cells with an effector memory phenotype (CD8ET), and NK recruiting cells (NKR). Sensitivity analyses and pleiotropy assessments excluded the bias due to weak instruments and linkage disequilibrium. Additionally, the associations were supported by strong evidence of colocalization with CRC. The expression of FHL3 in normal tissue was higher than that in the tumor tissue at the mRNA expression and single-cell levels, and low mRNA expression of FHL3 was associated with shorter survival time in CRC patients. Conclusions Our study provides a novel perspective on the potential causal link between the low expression of FHL3 gene and the risk of CRC. More evidence is essential to further reveal the biological mechanisms underlying this association.