Physical inactivity is known to be a risk factor for several noncommunicable diseases and has a high prevalence in today’s society. Therefore, it is crucial to understand the psychological factors associated with physical activity (PA). Recent developments in the field of ambulatory assessment and technological advances are promising to enhance our understanding of this relationship by analyzing longitudinal data within- and between-persons. These analyses can reveal important factors to design behavior change interventions to enhance PA. Therefore, this study used an ecological momentary assessment during the 3‑week intervention period in the SMARTFAMILY2.0 trial and aimed to investigate whether valence, calmness, energetic arousal, and sleep quality predict daily steps and moderate to vigorous PA. Overall, 49 adults (35–60 years) and 40 children (5–19 years) were included in this analysis and self-rated their mental state within our smartphone application while also wearing a hip-worn accelerometer for 21 consecutive days (996 days included) during the intervention period. Multilevel analyses were conducted to predict daily PA while considering covariables (e.g., child/adult and non-wear time) both within- and between-persons. The results indicated that higher than average ratings of a person’s valence and energetic arousal on one day predicted increased PA while higher than average calmness predicted decreased PA at the same day within this person. Sleep quality and between-person effects of the affective states showed no clear associations to PA. Overall, these results showed that within-person associations of valence, calmness, and energetic arousal should be considered when designing PA interventions for both children and adults. The influence of sleep quality, as well as between-person effects, should be further explored by future studies.