Abstract Predicting recurrence among early-stage hormone receptor-positive human epidermal growth factor receptor-negative breast cancer (HR+/HER2− BC) is crucial for guiding adjuvant therapy. However, studies are limited for patients with low recurrence risk. HR+/HER2− early-stage (T1-2N0-1) invasive BC patients who received definitive surgery and followed by endocrine therapy from four independent medical centers were included in this retrospective study. Patients from center 1 were used as derivation cohort, while those from other centers were combined as an external test cohort. A deep learning prognostic model, HERPAI, was developed based on Transformer to predict risk of invasive disease-free survival (iDFS) utilizing clinical and pathological predictors. The model performance was evaluated using C-index for the overall population and subgroups. Threshold for selecting 5-year recurrence risk > 10% was determined. Hazard ratio (HR) was estimated between risk groups for iDFS. A total of 6340 patients were included, of whom 5424 were assigned to the derivation cohort (training and validation [N = 4882] and internal test cohort [N = 542]), while 916 patients were utilized as external cohort. HERPAI yielded a C-index of 0.73 (95% CI 0.65–0.81), 0.73 (95% CI 0.62–0.85), and 0.68 (95% CI 0.60–0.77), in the validation, internal, and external test cohort, respectively. Consistent performances were observed for pre-specified subgroups. High-risk patients were associated with an increased risk of recurrence for validation (HR, 2.56 [95% CI 1.25–5.22], P = 0.01), internal test (HR, 2.52 [95% CI 0.97–6.57], P = 0.06) and external test (HR, 1.94 [95% CI 1.00–3.74], P = 0.049) cohort, respectively. HERPAI was a promising tool for selecting vulnerable early-stage HR+/HER2− BC patients who were at high-risk of recurrence. It could facilitate the prioritization of patients who may benefit more from escalating adjuvant treatment.