Mixed reality (MR) techniques enable physicians to interact with 3D virtual medical holograms and the real-world environment simultaneously, providing an intuitive understanding of the positional relationships between organs, blood vessels, and lesions within a patient's body. MR technology with head-mounted display provides a revolutionary tool in assisting surgical planning, intraoperative referencing, and navigation of minimally invasive surgery. However, the real-time motion of patient 3D model reconstructions and objective evaluation are still great challenges for MR-based simulation and navigation. We are developing learning-based methods for the 3D model of the patient’s anatomy and quantitative training evaluation. Tasks we focus on include: image segmentation for various anatomical structures and the tumor target, surgical instrument tracking, and multi-modal qualitative assessment strategy for objective evaluation in training.