Abstract Background Compelling evidence has established a strong link between the gut microbiota and host reproductive health. However, the specific regulatory roles of individual bacterial species on reproductive performance are not well-understood. In the present study, Jinhua sows with varying reproductive performances under the same diet and management conditions were selected to explore potential mechanisms on the intricate relationship between the gut microbiome and host reproductive performance using 16S rRNA sequencing, metagenomics and serum metabolomics. Results Our findings revealed that the KEGG pathways for base excision repair and DNA replication were enriched, along with gene-level enhancements in spore formation, in sows with higher reproductive performance, indicating that the gut microbiome experiences stress. Further analysis showed a positive correlation between these changes and litter size, indicating that the host acts as a stressor, reshaping the microbiome. This adaptation allows the intestinal microbes in sows with high reproductive performance to enrich specific serotonin-related bacteria, such as Oxalobacter formigenes, Ruminococcus sp. CAG 382, Clostridium leptum, and Clostridium botulinum. Subsequently, the enriched microbiota may promote host serotonin production, which is positively correlated with reproductive performance in our study, known to regulate follicle survival and oocyte maturation. Conclusion Our study provides a theoretical basis for understanding the interactions between gut microbes and the host. It highlights new insights into reassembling gut microbiota in sows with higher litter sizes and the role of serotonin-related microbiota and serotonin in fertility. Graphical Abstract