An Open-Label, Multicenter Study to Investigate the Safety, Tolerability, Pharmacokinetics, and Preliminary Efficacy of XL309 (ISM3091) as Single-Agent and Combination Therapy in Patients With Advanced Solid Tumors

Key Inclusion Criteria: 1. Capable of understanding and complying with protocol requirements. 2. Male or female aged 18 years or older. 3. Eastern Cooperative Oncology Group performance status 0 or 1. 4. Adequate bone marrow and organ function. 5. Participant-disease Characteristics Dose-Escalation Stage Single Agent and Combination: a) Participants whose tumor progressed on, or who were intolerant to standard therapy, have a disease for which no therapy exists or are not a candidate for these therapies, and have one of the following cancers: i. Histologically confirmed locally advanced/metastatic human epidermal growth factor receptor-2 (HER2)-negative breast cancer, with deleterious or suspected deleterious breast cancer gene (BRCA)1/2 alteration. ii. Histologically confirmed locally advanced/metastatic high-grade serous ovarian cancer (HGSOC), including primary peritoneal cancer (PPC) and fallopian tube cancer (FTC). iii. Histologically confirmed locally advanced/metastatic CRPC, with deleterious or suspected deleterious BRCA1/2 alteration. iv. Histologically confirmed locally advanced/metastatic pancreatic cancer with deleterious or suspected deleterious BRCA1/2 alteration. v. Locally advanced/metastatic tumors with deleterious or suspected deleterious germline or somatic homologous recombination repair (HRR) mutation or homologous recombination deficiency (HRD) phenotype. Cohort-Expansion Stage Single Agent and Combination: b) HER2-negative breast cancer cohort: participants with histologically confirmed locally advanced/metastatic (HER2)-negative breast cancer with alterations in select HRR genes. c) Platinum-sensitive HGSOC cohort: participants with histologically confirmed locally advanced/metastatic HGSOC, including primary peritoneal cancer (PPC) and fallopian tube cancer (FTC), with positive HRD result using an approved diagnostic, and/or alterations in select HRR genes. d) mCRPC cohort: participants with metastatic, castration-resistant adenocarcinoma of the prostate with alterations in select HRR genes. e) HRRm advanced solid tumors cohort: participants with locally advanced/metastatic tumors with alterations in select HRR genes. For all participants with solid tumors: 6. Participants in the Cohort-Expansion Stage must have at least 1 measurable target lesion. 7. Recovery to baseline or ≤ Grade 1 CTCAE v5 from AE(s) related to any prior treatments. Key Exclusion Criteria 1. Prior anticancer treatment including: 1. Small molecule-targeted therapy \< 5 half-lives from first dose of study treatment, or 3 weeks (whichever is shorter). 2. Any antibody therapy \< 5 half-lives from first dose of study treatment (or 4 weeks since last therapy, whichever is shorter). 3. Chemotherapy with nitrosoureas or mitomycin C \< 6 weeks from first dose of study treatment. Other chemotherapy \< 3 weeks prior to first dose of study treatment. 4. Radiation therapy (including radiofrequency ablation) \< 1 week prior to initiation of study treatment. Participants with clinically relevant ongoing complications from prior radiation therapy are not eligible. 2. Known brain metastases or cranial epidural disease unless adequately treated with radiotherapy and/or surgery (including radiosurgery) and stable for at least 4 weeks before first dose of study treatment. 3. History of hypersensitivity to any excipient of XL309, or history of allergic reactions attributed to drugs with a similar chemical or biologic structure or class to XL309. 4. Lactating or pregnant females. 5. Clinically relevant cardiovascular disease. 6. Known history of myelodysplastic syndrome. 7. Other severe, acute, or chronic medical condition or laboratory abnormality that may increase the risk associated with study participation or study drug administration, or that may interfere with the interpretation of the study results, and in the judgment of the investigator, would make the participant inappropriate for the study. 8. Inability or unwillingness to comply with requirement for oral drug administration or presence of a gastrointestinal condition that would preclude adequate absorption of XL309. 9. Prior treatment with a ubiquitin specific peptidase 1 (USP1) inhibitor.