Phase II Single-Arm Study of Enfortumab Vedotin (EV) Plus Pembrolizumab in the Treatment of Locally Advanced or Metastatic Bladder Cancer of Variant Histology

Inclusion Criteria: * Male or Female * Age \>= 18 years * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%) * Metastatic disease or unresectable locally advanced disease * Histologically documented variant histology (nested, microcytic, micropapillary, lymphoepithelioma-like, plasmacytoid, giant cell, poorly differentiated, lipid-rich, clear cell, sarcomatoid) bladder cancer and non-urothelial bladder cancer of epithelial origin including squamous cell carcinoma and adenocarcinoma (urachal and non-urachal). Variant histology tumors and non-urothelial tumors of ureter, urethra, urachus, or renal pelvis are included. Patients with mixed cell type are eligible if the predominant histology (over 50%) is variant or non-urothelial. All histological classifications will follow the 2016 WHO Classifications. * Untreated or having received any number of lines of prior therapy * Tumor tissue samples must be available for submission prior to initiation of study treatment. If not, agree to undergo biopsy * Patients must have measurable disease as defined by RECIST criteria 1.1 as at least one lesion that can be accurately measured in at least one dimension (longest diameter of \>= 10 mm for non-nodal lesions or short axis of \>= 15 mm for nodal lesions) on CT scan, MRI * Patients must have adequate organ and marrow function, within 28 days of cycle 1 day 1, at the discretion of the investigator * The effects of study drugs on the developing human fetus are unknown. For this reason, female of child-bearing potential (FCBP) must have a negative serum or urine pregnancy test prior to starting therapy * FCBP and men treated or enrolled on this protocol must agree to use adequate contraception (hormonal or barrier method of birth control; or abstinence) prior to study entry and for the duration of study participation. Additionally, FCBP and male subjects should use effective contraception for 6 months after the last dose. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Male subjects must not donate sperm and female subjects must not donate ova from screening to 6 months after the last dose * A female of childbearing potential (FCBP) is a sexually mature woman who: 1) has not undergone a hysterectomy or bilateral oophorectomy; or 2) has not been naturally postmenopausal for at least 24 consecutive months (i.e., has had menses at any time in the preceding 24 consecutive months) * Completion of all previous therapy (including surgery, radiotherapy, chemotherapy, immunotherapy, or investigational therapy) for the treatment of cancer \>= 4 weeks before the start of study therapy * Life expectancy \> 12 weeks as determined by the Investigator * Willingness and ability of the subject to comply with scheduled visits, drug administration plan, protocol-specified laboratory tests, other study procedures, and study restrictions * Evidence of a personally signed informed consent indicating that the subject is aware of the neoplastic nature of the disease and has been informed of the procedures to be followed, the experimental nature of the therapy, alternatives, potential risks and discomforts, potential benefits, and other pertinent aspects of study participation Exclusion Criteria: * The neuroendocrine histology (small cell and large cell carcinomas) and non-epithelial bladder tumors (e.g. bladder sarcoma, carcinosarcoma, paraganglioma, melanoma, primary lymphoma, and lymphoepithelioma-like carcinoma) are excluded. * Patients who have had chemotherapy or radiotherapy within 4 weeks prior to entering the study or those who have not recovered from adverse events due to agents administered more than 4 weeks earlier \[i.e. ongoing clinically significant toxicity (grade 2 or higher with the exception of alopecia) associated with prior treatment\] * Patients who are receiving any other investigational agents or an investigational device within 21 days before administration of first dose of study drugs * History of allergic reactions attributed to compounds of similar chemical or biologic composition to the agents used in study * Patients with ongoing sensory or motor neuropathy grade \>= 2 * Prior treatment or enrollment in a study with EV or PD1/PD-L1 immune checkpoint inhibitor (including maintenance therapy) * Known uncontrolled diabetes mellitus with glycated hemoglobin (HbA1c) \>= 8% or HbA1c 7% to \< 8% with associated diabetes symptoms (polyuria or polydipsia) that are not otherwise explained * Active central nervous system (CNS) metastases * Excluding the primary tumor leading to enrollment in this study, any other active malignancy (except for localized prostate cancer, definitively treated melanoma in-situ, basal or squamous cell carcinoma of the skin, or carcinoma in-situ of the bladder or cervix) within the past 24 months * Currently receiving systemic antimicrobial treatment for active infection or high dose steroids (\> 10mg of prednisone or equivalent) * A FCBP who has a positive urine pregnancy test at baseline or within 72 hours prior to receiving first study dose. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Breastfeeding females * History of active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs), known hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive), known active hepatitis C virus (defined as HCV ribonucleic acid \[mRNA\] \[qualitative\] is detected) or tuberculosis * History of active keratitis or corneal ulcerations * History of allogenic tissue/solid organ transplant * Congestive heart failure New York Heart Association Class 3 or 4, unstable angina pectoris, serious cardiac arrhythmias within 6 months prior to first dose of EV/pembrolizumab * Other uncontrolled current illness including, but not limited to, cardiac arrhythmia or psychiatric illness/social situations that would limit compliance with study requirements