Abstract Background Plant invasion affects plant community composition, biodiversity, and nutrient cycling in terrestrial ecosystems, particularly in vulnerable ecosystems. As an invasive parasitic plant, Cassytha filiformis has caused extensive damage to the native vegetation of the Paracel Islands. However, the effects of C. filiformis invasion on litter decomposition and nutrient release in native plant communities remain unclear. We conducted an in-situ decomposition experiment in native plant communities on a coral island to explore the litter decomposition dynamics varying across enzyme activities, soil properties and C. filiformis invasive degrees. Results The mass loss of litter was determined during the decomposition process. The data showed that litter mass loss under severe invasion was significantly lower than in uninvaded sites after nine months of decomposition. The invasion of C. filiformis accelerated the nitrogen release and lignin decomposition with increased litter quality and polyphenol oxidase activity. Besides, soil phosphorus availability and potassium content also induced the oxidase activity. Meanwhile, the decomposition of litter organic carbon was delayed because β-1, 4-glucosidase activity was low in the first six months. Besides, peroxidase activity maintained a high level in invasive plots, indicating that the residues of C. filiformis may have allelopathy. Conclusion Our results suggested that the invasion of C. filiformis accelerated litter mass loss and element release on coral islands by regulating litter quality and enzyme activity. However, the short-term rapid litter decomposition may result in nutrient loss, which is not conducive to the growth of native plants.