Abstract The deformation behavior of the roof and floor strata during coal seam mining plays a critical role in the development of mining–induced fractures and is an intrinsic factor in the prevention and control of coal and rock dynamic disasters, such as mine water hazards. This study investigates the critical values of the rock failure angle displacement based on the deformation performance of the strata through both laboratory and field experiments. A method for assessing the strata failure condition using the differences in deformation performance is proposed. The results indicated that when the angular displacement was greater than the critical value, the rock layer was destroyed, and the critical value of angular displacement was related to lithology, thickness and cross–sectional area. The critical value of angular displacement of rock strata with high strength was larger, the critical value of angular displacement of sandstone was 0.0552 ° on average, and that of limestone was 0.0818 °. The angular displacement of rock strata with small strength under unit load was larger, resulting in greater deformation. The angular displacement of sandstone under unit load was 5.45 × 10− 6 ° / N, and that of limestone was 4.41 × 10− 6 °/ N. The critical value of angular displacement of rock strata with larger thickness and cross–sectional area was smaller. The critical values of angular displacement of sandstone and limestone with 50 mm thickness were 0.0516 ° and 0.0774 °, respectively, and the critical values of angular displacement less than 30 mm thickness were 0.0552 ° and 0.0818 °. The critical values of angular displacement of sandstone and limestone with a cross–sectional area of 150 mm × 50 mm were 0.0483 ° and 0.0720 °, respectively, which were less than the critical values of angular displacement of 120 mm × 40 mm of 0.0516 ° and 0.0774 °. The difference between the angular displacement (θ i) and the critical value (θ 0) under unit stress leads to the time sequence difference between horizontal and vertical fractures and the ‘anisotropic seepage’ law of groundwater. It was more convenient to use the critical value of angular displacement under unit stress as the criterion for judging the deformation and failure of rock strata than the traditional mechanical criterion. The research results have important practical significance for judging the failure of mining strata, analyzing the movement law of mining groundwater, and calculating the depth of water–conducting fracture zone and floor failure zone.