Funded by William T. Grant Foundation in 2013-2015, the purpose of Afterschool Program Practices Tool (APT) II Study was to develop and test drive a multi-pronged online reliability training designed to improve rating accuracy for youth program observations. In this study, the research team found preliminary evidence that trainees found it somewhat more difficult to accurately rate video clips (a) when behaviors to be rated reflected the middle range of the APT scale, (b) that featured programs for younger, elementary-aged children, (c) that were the lengthiest, and (d) that contained behaviors that could be interpreted differently based on one’s culture, background, or experience. An understudied yet crucial source of measurement variance within observational tools is whether the “gold standard” ratings that account for whether a trained rater passes certification of reliability has cultural biases that would unfairly privilege some groups of people with certain cultural vantage points over others. As a supplement to APT Validation Study II, the research aims for Validation Study III are to (1) generate master scores for video clips of youth program observations without cultural bias, (2) create more tailored and targeted online training and anchor systems, and (3) eliminate significant differences in certification passing rates between groups with different cultural vantage points (i.e., Black vs. White raters, urban vs. non-urban program experiences). Products from this supplemental study include (a) more culturally inclusive Guide to Activity anchor ratings, (b) guide to master scores, (c) “range finding” online training tools, and (d) culturally informed master scores for online assessments used for APT reliability certification purposes.