Questions on justice of earnings are regularly fielded in large-scale surveys but insights into the role of response formats on measures of the justice of earnings are missing. This problem is illustrated by the German Socio-Economic Panel Study (SOEP), which, in 2017, changed its question on the justice of one’s own earnings from a binary response scale to an 11-point scale. Meanwhile, the share of respondents evaluating their earnings as just dropped considerably, leaving unclear how methodological and substantive effects are intertwined. Addressing this gap, we analysed a survey experiment in the 2016 Innovation Sample of the SOEP (SOEP-IS). In a split-ballot design, 2562 employed SOEP-IS respondents were randomly allocated to one of two experimental groups: receiving either the binary scale or the 11-point scale. Our results show that a lower share of respondents evaluated their earnings as just in the 11-point scale condition. However, follow-up questions on the just amount of earnings were unaffected by the question format. We conclude that it is crucial for researchers investigating justice evaluations of one’s own earnings to account for these measurement effects as well as for practitioners to carefully document and test the effects of changes in response format.