Accurate indoor 3D models are essential for building administration and applications in digital city construction and operation. Developing an automatic and accurate method to reconstruct an indoor model with semantics is a challenge in complex indoor environments. Our method focuses on the permanent structure based on a weak Manhattan world assumption, and we propose a pipeline to reconstruct indoor models. First, the proposed method extracts boundary primitives from semantic point clouds, such as floors, walls, ceilings, windows, and doors. The primitives of the building boundary are aligned to generate the boundaries of the indoor scene, which contains the structure of the horizontal plane and height change in the vertical direction. Then, an optimization algorithm is applied to optimize the geometric relationships among all features based on their categories after the classification process. The heights of feature points are captured and optimized according to their neighborhoods. Finally, a 3D wireframe model of the indoor scene is reconstructed based on the 3D feature information. Experiments on three different datasets demonstrate that the proposed method can be used to effectively reconstruct 3D wireframe models of indoor scenes with high accuracy.