The effects of family backgrounds on educational decisions have been well studied both in Germany and internationally. However, one aspect that has not yet been thoroughly investigated is the influence of contextual factors related to the residential and school environment of children and parents. We ask how neighbourhood, school and family contexts affect educational decisions, focusing on the most consequential sorting process in the German education system, the transition from primary to lower secondary school. To capture this transition, we analyse not only its outcome but also two critical precursors: parental educational aspirations in primary school and teachers’ recommendations for the academic track. Controlling for prior academic achievement allows us to quantify the secondary effects these contexts exert. Based on data of NEPS Starting Cohort 2, supplemented with geocoded residential address data on a 1000x1000-meter grid, our results show that neighbourhood composition, school composition and family backgrounds are tightly interlinked, pointing to residential segregation, which is reflected in school composition. While effects of neighbourhood educational and ethnic composition are found when they are introduced solely, they disappear as soon as school context and family backgrounds are accounted for. In contrast, school composition does still matter in the full models. In particular, students in schools with higher shares of immigrant background students tend to have better outcomes, on average. The most important factor directly influencing transition outcomes is individual family backgrounds. We conclude that despite the lack of direct effects, the neighbourhoods where children live do matter because their composition is reflected in schools and they might have influenced their parents over time. To equalize children’s chances to access the academic track it seems to be as important to support individual families from early on as to tackle residential segregation.