With the rapid development of digital medical imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound imaging (US), and digital pathology, advanced imaging technology has played an essential role in solving the challenges of tumor diagnosis and treatment for clinicians. However, malignant tumor is a complex and heterogeneous disease; it varies significantly from patient to patient or from different tumor sites. A single imaging method has its limitations in resolution, sensitivity, and contrast due to the physical, chemical, and biological characteristics of varying imaging principles. This project aims to develop novel multi-modal fusion learning methods for segmentation and classification, leveraging across-scale and across modality information to create repeatable, reproducible, interpretable tumor characteristic models in clinical diagnosis and treatment.