We develop an additive Cox proportional hazard model with time-varying covariates, including spatio-temporal characteristics of weather events, to study the impact of weather extremes (heavy rains and tropical cyclones) on the probability of mortgage default and prepayment. We compare the survival model with a flexible logistic model and an extreme gradient boosting algorithm. We estimate the models on a portfolio of mortgages in Florida, consisting of 69,046 loans and 3,707,831 loan-month observations with localization data at the five-digit ZIP code level. We find a statistically significant and non-linear impact of tropical cyclone intensity on default as well as a significant impact of heavy rains in areas with large exposure to flood risks. These findings confirm existing results in the literature and also provide estimates of the impact of the extreme event characteristics on mortgage risk, e.g. the impact of tropical cyclones on default more than doubles in magnitude when moving from a hurricane of category two to a hurricane of category three or more. We build on the identified effect of exposure to flood risk (in interaction with heavy rainfall) on mortgage default to perform a scenario analysis of the future impacts of climate change using the First Street flood model, which provides projections of exposure to floods in 2050 under RCP 4.5. We find a systematic increase in risk under climate change that can vary based on the scenario of extreme events considered. Climate-adjusted credit risk allows risk managers to better evaluate the impact of climate-related risks on mortgage portfolios.