Warped perceptions: recent brain scan explorations of how social stereotypes affect vision.


*All illustrations are adapted in their original form, or with my own modifications, from the publication by Stolier & Freeman (2016), unless stated otherwise. All scientific studies I mention in the text can be accessed by clicking on the links provided.

It’s undeniable that social stereotypes routinely bias how humans perceive others, at least on an implicit level. While such preconceptions often influence our interpretations of behaviours (eg. ‘How typical of a woman to get emotional in this situation’), evidence from psychology laboratories indicates that stereotypes might fundamentally shape how our visual system processes individuals’ faces in the first place, before any behaviour that might expose them to social judgement even takes place. To put a cherry on top of this cake, researchers from New York University have recently published evidence from fMRI brain scans supporting the idea that the way in which the brain processes faces might literally be warped by the social stereotypes we hold. Aside from accounting for well-known racial and gender biases, this experiment offers some insight into more cryptic perceptual phenomena that are not as obviously explained by common prejudices. Let’s take a look at what these are before we dive into the experiment itself.

Classical versus ‘entangled’ stereotypes

It might come as no huge surprise that psychological studies have found that an individual’s apparent race appears to provoke automatic assumptions about their likely emotional expression. One piece of evidence for this stems from an experiment conducted at Northwestern University, in which Caucasian participants were shown computer-generated faces gradually morphing from a neutral to hostile expression and asked to identify the point at which each face displayed an unambiguous level of hostility. The researchers found that participants with high levels of implicit racial bias, measured using several tests, identified the onset of unequivocal hostility significantly more quickly for Black, as opposed to Caucasian, faces. Findings such as these have also been reported by researchers studying Caucasian students in the United Kingdom. On a similar note, maleness has also been found to be perceptually associated with anger, while femaleness is with happiness. These findings are quite in line many widespread social prejudices, such as the common assumption that women are more friendly and docile than men, or the colonial stereotype of the aggressive African-American. But other perceptual phenomena appear to be less consistent with this account.

For several decades now, psychologists have had evidence that race and gender appear to be randomly entangled during the perceptual processing of faces. In one experiment showing this effect, participants were shown images of gender-ambiguous faces generated using race-morphing computer software that endowed the faces with varying degrees of Black, Caucasian and Asian features. When individuals were asked to classify the gender of each face, it turned out that such judgements were strongly influenced by the entirely irrelevant dimension of race. Faces with the strongest Black features were significantly more likely to be identified as male, while Asian faces were, on the converse, more likely to be categorised as female. It is important to note that, before these computer-generated faces were endowed with various ethnic features, they all started out with an equal level of gender-ambiguity. In light of this, the discrepancy between ‘male’ and ‘female’ classifications given to Black and Asian faces implies that ethnicity had a significant impact on how the participants’ visual systems made sense of the gender-ambiguity. Thus, it seems that perceiving a face as belonging to a particular race can generate automatic expectations of the person’s likely gender, to an extent that actually biases what people end up seeing. Needless to say, these preconceived connections between race and gender aren’t actually rooted in any physical reality. They also do not appear to be rooted in social stereotypes – as far as I am concerned, there are no explicit prejudices directly linking race and gender. So how can Asian ethnicity be so strongly associated with femaleness that people are actually more likely to see a gender-ambiguous face as female if it’s Asian?

Researchers suggest that this entanglement of race and gender can emerge due to overlap that might occur between our stereotypical representations of various social groups. If we examine the Asian-Female link through the lens of this argument, we might consider that Western society stereotypically depicts individuals of East Asian descent as docile and unthreatening – a set of assumptions that heavily overlaps with the female stereotype. As a result of the fact that the stereotypes of these two unrelated social groups share characteristics, the ways in which our brains represent these groups become fundamentally entangled. Thus, when we make a visual judgement on an individual’s ethnicity and our brains begin retrieving the contents of a particular racial stereotype, the by-product of this might be that the gender stereotype sharing some of the same content also becomes activated. Ultimately, racial categorisation could actually bias our visual system to perceive an individual as belonging to one gender as opposed to another.

Before jumping into how the researchers at NYU pinned down the effects of stereotyping in the brain, let’s have a look at how they measured such prejudices in the first place.

Measuring perceptual prejudice

Participants performed a mouse-tracking task (shown below) with a massive range of computer-generated faces combining different ethnicities, emotional expressions and genders. Across hundreds of trials, they were shown images of faces and asked to make rapid categorical decisions about each one, such as whether it was male or female, happy or angry, Black or White. (Of course, the existence of categorical distinctions between ethnicities, genders and emotional states is entirely debatable, but given that human societies are largely built on categorical premises, it makes sense to interrogate their effects on brain and behaviour.)

Neuro1.pngAs participants moved the cursor across the screen to indicate the correct answer, its coordinates were being tracked 70 times per second. This precise record of the mouse trajectory that occurred during the course of the perceptual decision allowed the researchers to gauge the degree to which race, gender, and emotionality were entangled during perceptual processing.

How can mouse-tracking achieve this? The idea behind it is that the cursor trajectory can measure how easy it was for a person to make a particular decision – and thus, what types of perceptual expectations they might have had. When an individual is asked to make a judgement that agrees with expectations, it’s presumably a no-brainer, driving the hand directly to the correct answer on top of the screen. However, if the alternative option is actually more consistent with one’s intuitions and deep-seated expectations, the hand might temporarily hesitate, initially moving the cursor to this competing option before swerving towards the answer. To take an example – if you are asked to classify the face of a smiling Black man as ‘happy’ or ‘angry’, having a strong subconscious stereotype that Black men are aggressive might automatically drive your hand towards the ‘angry’ option first, before you thoroughly process the image and identify its positive emotional expression. Thus, the degree to which the mouse deviates from the ideal path can tell us something about how strongly the two response options on the screen correspond to the stereotypes an individual might hold about the ethnic features or gender of the face they are looking at. Importantly, this movement is so rapid that it is assumed to tap into deeply ingrained perceptual assumptions not entirely accessible to conscious thought.

To better understand this, let’s examine the illustration below, which contains the mouse trajectories of two hypothetical individuals given an image of an Asian man and asked the same question: is this face angry or happy? The goal of this trial is to measure how compatible the concepts of happiness and anger are with the stereotypes that the individuals might have about the features of this face – namely, its ethnicity and gender. This is captured by an index of similarity.


When these similarity scores are collected in large numbers from each participant, they enable researchers to quantify how strongly various ethnic, gender and emotional categories are connected in each participant’s perception. To explain this, the illustration below shows mouse trajectories taken from three separate trials in which an individual is asked to decide if a face is angry or happy.


When we consider the pair of similarity scores shown above, the happy Asian man is perceptually more similar to the happy Caucasian woman than to the angry Asian man. But things don’t end here. Next, researchers proceed to calculate similarity scores for each of these faces with respect to ethnic and gender categories by asking participants other questions, such as: Is this face Asian or White? Is this face female or male? Taking into account all the similarity scores they obtain for each face, the experimenters can go on to calculate a single value that describes how subjectively similar these three faces are to each other. Essentially the question is: Neuro4.pngUltimately, the goal is to estimate how subjectively similar each of these faces is to each other. In this recently published experiment, the researchers did this for every single face they used in their study. Below is an example matrix of subjective similarities taken from one individual participant.neuro5

Here comes the icing. The massive matrix that documents the subjective similarities between faces differing in terms of expression, gender and ethnicity is ‘refined’ to produce a different kind of matrix: one that describes the subjective similarities of the underlying social categories, through the eyes of the participant. This matrix directly answers questions such as: how strongly does this individual’s perception associate Asian ethnicity and maleness? How about femaleness and anger? How about Black ethnicity and happiness? In essence, it summarises one person’s stereotypes and the tangles between them in one picture. The image below explains this in greater detail.


For this participant, Asian ethnicity is subjectively quite dissimilar to anger (1), while Black ethnicity is subjectively quite similar to it (2). Whiteness appears to have a more neutral association with this emotional state (3). If you take a closer look at this matrix, it becomes apparent that ethnicity and gender are also subjectively intertwined. For instance, Black ethnicity is subjectively very similar to the male concept, while Asian ethnicity is quite dissimilar to it. This is a good example of category entanglement, since there are no explicit stereotypes linking race and gender, and yet these two dimensions appear to be bound because their stereotypes share some content.

Identifying effects of social stereotyping in the brain

To examine whether an individual’s pattern of social stereotypes had an impact on their brain, the researchers used fMRI brain scans to record patterns of brain activity as their participants looked at images of different faces. Recording such patterns involves ‘dividing’ the brain into tens of thousands of voxels – tiny cubes of brain tissue that contain millions of neurons each – and examining levels of activity across all of them.


So how do we test if the activity pattern of any brain region corresponds to an individual’s profile of social stereotypes? In essence, we need to look back at the person’s matrix of subjective similarities and ask a variety of questions:


Using this method, the researchers performed a ‘searchlight’ across the brain and found two regions where activity patterns that occurred when an individual looked at different faces corresponded to the pattern of subjective similarities that was extracted when that individual performed the mouse-tracking task. One of these regions was the orbitofrontal cortex (shown below).


The neurons in this area are thought to contain information about an individual’s subjective preferences, as numerous studies have found that orbitofrontal cells show higher levels of activity when individuals consider options they personally value more (eg. chocolate bar) as opposed to less (eg. celery stick). The finding that the activity patterns of an area considered to represent subjective preferences to some extent reflect an individual’s social stereotypes is perhaps not entirely surprising. Social prejudices can in many ways be seen as synonymous with preferences, as racially prejudiced individuals often value those belonging to one ethnic group above or below others.

Perhaps more surprisingly, the pattern of an individual’s subjective similarities was also reflected in the activity of the fusiform gyrus – an area considered to be directly involved in the visual processing of faces. This region, found both in humans and monkeys, contains neurons that are preferentially activated by the sight of faces, giving rise to the hypothesis that it is critical for our ability to recognise them. Indeed, human patients with damage to the fusiform gyrus or some surrounding brain areas tend to struggle with face identification, often relying on individual features to help them recognise others (eg. the fact that someone has red hair, or has a particular mole on their right cheek). So what does the finding that activity patterns in this region reflect an individual’s stereotypes really mean? To take an example, it means that in those individuals who view Asian ethnicity and femaleness as subjectively similar, the brain region involved in the visual recognition of faces actually treats Asian faces and female faces quite similarly as well – i.e. produces highly similar patterns of activity when the individual sees faces from these two categories. The researchers suggested that this effect might be the result of the orbitofrontal cortex communicating socio-cultural information about stereotypes to the visual region in order to bias face recognition in a direction that is consistent with social expectations (though this remains to be directly tested).

Perceptual warping in accordance with stereotypes might have been previously useful to humans when they lived in small and competitive hunter-gatherer communities – in these times, biased perception could have made people quicker at detecting and responding to hostility in the face of a rival group member. However, in our current environment such warping can only be corrosive, so perhaps the evidence serves as a reminder to be aware that our biases might be more deep-seated than we believe.


2 thoughts on “Warped perceptions: recent brain scan explorations of how social stereotypes affect vision.

  1. Instantly made into a generalized article on race and gender. An african black person knows exactly which tribe another african black person comes from and has most likely interpreting interpretation of facial expressions. Similarly in Iceland where women have an equal to men position, reactions towards women are not anyway like in most other countries. And certainly on race and gender no other country compares to one of the most inequal, biased country like the USA.


  2. Thanks for commenting! My take on gender / racial stereotypes is that they are not intrinsic, and instead most likely taught and culturally transmitted. This obviously means (this is where I completely agree with you) that both behavioural and neuroimaging evidence about stereotyping does not apply across the entire world. Having said that, I firmly believe it applies to much of the Western world. We can be pretty certain that post-colonial nations with a long history of racially-based exploitation and segregation will have stereotypes of this nature in their populations. The same probably goes for many Western countries with no explicit history of colonialism, but who are currently dealing with an influx of migrants from other ethnic groups – no doubt there, racially-based tensions are present. As for gender stereotypes – the case of Iceland, where women are legislatively equal to men in society, does not necessarily mean that people there still don’t have some deep-seated perceptions of what constitutes a female attribute. They’ve had hundreds of years living with such stereotypes, and there are of course questions about whether enforcing equality from a legal perspective truly wipes out all personal prejudices. At the end of the day, my intention was not necessarily to imply that the entire world harbors the stereotypes that are examined in this study. The experiment used these commonly held stereotypes as a ‘model’ to study stereotyping more generally, and as such the findings apply irrespective of the exact nature of the stereotype a person holds. At the end of the day, we might agree that populations from every place in the world have some forms of stereotypes.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s