Even when people hold little prejudice themselves, expectations about how members of other groups perceive them may negatively influence interracial relations. In four pre-registered experiments, each using a full intergroup design with Black and White participants, we show that people infer negative meta-attitudes from out-group members whose appearance is phenotypically prototypical, which in turn leads to less favorable orientations towards intergroup contact, independent of personal attitudes. In Experiment 1, Black Americans, but not White Americans, perceived that more phenotypically prototypical out-group members held less favorable meta-attitudes, and this explained less favorable contact orientations. In Experiment 2, this pattern emerged for both groups of participants and was particularly pronounced among individuals higher in stigma consciousness. Experiment 3 replicated Experiment 2 with representative samples and demonstrated that the effect of phenotypic prototypicality was more pronounced among participants reporting greater previous rejection by the out-group. With few exceptions, participants in the experiments also perceived phenotypically prototypical in-group members as having less positive meta-attitudes and participants showed less favorable contact orientations toward these in-group targets. An internal meta-analysis supported the robustness of the findings in the first three experiments. In Experiment 4, direct evidence for the causal effect of the mediator meta-attitudes on orientations toward contact with in- and out-group members was obtained. In all studies, effects held controlling for participants’ general intergroup attitudes and experiences, demonstrating the unique role of meta-attitudes in shaping intergroup relations. We discuss our results in light of previous research, highlight social implications, and suggest future directions.