Primate vocal communication, including human speech, is a multisensory function—perceived through both the visual and auditory channels. The existence of multiple nodes in the cortical network that integrate faces and voices suggests that they may be interacting and influencing each other during natural communication. One mechanism for establishing functional interactions between separate neuronal populations is transient coupling in the form of temporal correlations. To test hypotheses that suggest that multisensory responses to communication signals in auditory cortex are influenced by visual inputs from the superior temporal sulcus (STS), we record single neurons and local field potentials from both structures concurrently. This allows us to investigate how signals from these structures influence each other through their spikes and oscillations. A related project seeks to link these sensory interactions with the motor system. Rhesus monkeys (like humans) predominantly focus on the eye region versus the mouth when viewing vocalizing conspecifics. Furthermore, fixations on the mouth are tightly correlated with mouth movements. There is a robust, three-node cortical circuit that likely mediates this sensorimotor behavior: the frontal eye fields of the frontal lobes, the STS, and the auditory cortex. These three regions are densely interconnected and we are recording from all three structures concurrently to examine this network’s activity during natural acts of communication.

In the traditional framework, vocalizations are a priori features of the acoustic environment that the auditory system must learn or evolve to process in order to generate an adaptive response. However, beyond the external niche, there may be other constraints that shape the evolution of vocal structure. One such constraint is the structure of the brain itself. The basic pattern of neocortical operations is conserved throughout the mammalian lineage, as evident in the ubiquity of neocortical oscillations in stereotypical frequency ranges. As some of these oscillations are thought to chunk sensory information in manageable segments, one hypothesis is that the structure of vocalizations should be matched to the frequency structure of neocortical oscillations. We are currently exploring to what degree the acoustic structure of vocalizations are designed to exploit the pre-existing physiological properties of the neocortex and what influence this may have on vocal recognition.

One of the practical problems with neurophysiology on behaving animals is the highly artificial context and structure of the experiments. One does not capture what the brain does under natural circumstances when the monkey is ‘passively’ viewing brief clips of social stimuli presented in a random order and in isolation, nor does the brain operate by averaging across trials. In an effort to move away from this standard paradigm and towards a more naturalistic context, we are pioneering an interactive playback paradigm in which the subject monkey interacts with a synthetic monkey agent on a video screen. The synthetic agent has a suite of eye gaze trajectories, facial expressions and vocal expressions. By measuring eye movements and facial expressions (via electromyography), we can use the subject monkey’s reactions to trigger the behavior of the synthetic agent. The synthetic agent can be made ‘dominant’ or ‘subordinate’ by its reactions and appearance. The sequential constraints in social communication by the subject and synthetic agent can be modeled after stochastic processes that occur in dyadic interactions of monkeys in the wild. Ultimately, the use of the synthetic agent will allow us to capture neurophysiologically how monkeys base their future social behaviors upon their memory of preceding events in their social interactions with a particular agent.

One long standing puzzle in the social cognition of primates (including humans) is how individuals can acquire the exquisite social knowledge—who belongs to which group, who is dominant, who is subordinate—required to behave adaptively. Can a few simple rules explain the complexity of a primate’s social knowledge or is there a special innate mechanism (as proposed by Cheney and Seyfarth (2007)) that facilitates our capacity to keep track of social relationships? We recently tested the former hypothesis by asking whether faces could be grouped by a statistical learning mechanism similar to that demonstrated for visual object grouping (Fiser and Aslin, 2001; Fiser and Aslin, 2002), and whether voices could influence this unsupervised and unconscious form of learning. Statistical learning is a mechanism that allows us to use probabilistically-defined patterns to structure our sensory environment. We tested the hypothesis that faces are learned differently from shapes, and that sound can influence the learning of these visual elements. Our data suggest that faces and shapes are learned equivalently when presented in spatial arrays and temporal sequences. Coupling faces with voices or shapes with synthetic sounds did not enhance visual learning, nor were sounds learned in parallel. Unexpectedly, when faces and voices were uncoupled, visual learning of faces was impaired. In contrast, when shapes and synthetic sounds were uncoupled, subjects showed evidence for parallel learning. These results suggest that the mechanisms mediating visual statistical learning may be superficially domain-general, but can be influenced in different ways by the auditory system depending on the nature of the elements and their real world associations. The next step is to replicate the experiment using monkeys as subjects which would allow us to explore the neural mechanisms.

Primate Neuroethology Laboratory, copyright 2007. All rights reserved. Design by Sunny Khemlani.