It is by now well known that the physical living environment has a major impact on people's life, but the neural structures involved in this relationship remain to be explored. Most studies investigating this relationship only focus on single environmental predictors. In order to understand how the multitude of factors constituting the living environment relate to brain structure we used data from the UK Biobank (n = 21,094; age Mean = 63.35 years; SD = 7.46; range = 45–82) to examine how individuals' immediate characteristics around the home address (e.g., green space; air pollution in the neighborhood) are associated with hippocampal volume, a brain region known to be highly plastic. We accounted for common demographic factors that have been shown to be associated with brain structure and known factors such as sex, income, education, and age. We made use of an analytical paradigm based on the feature importance estimation and recursive feature elimination with decision tree ensembles as well as linear regression analysis. Results identified a subset of environmental measures (e.g., pollution, green space, noise) most strongly associated with hippocampal volume across adulthood. Findings highlight the importance of the environment for individuals' brain structure.