For today, much of what we have been hearing about is context, context in perception. Neuroscientist are usually interested in is how context affects what we see. But today I am going to focus on the question of why context matters. The effect of context is fundamental to everything that the brain does even what is most basic to the brain … seeing colour. One of the examples I am going to show here is a well known color contrast illusion vision. As a neuroscientist I get very excited about very simple effects and why this comes about.
In order to answer the question why context matters, we have to first understand the basic challenge of vision, what the visual brain evolved to solve. That fundamental problem … or challenge .. is in the stimuli, that the pattern of lights that fall upon the eye are infinitely ambiguous. Any pattern that falls on the eye could mean anything. And the reason why it could mean anything is that light conflates multiple attributes of an object. Take a surface and put under different lights. The light coming from that surface will constantly change. And yet to be useful, the visual brain has to interact with the surface according to its differences and similarities with other objects in the world. If the light falling on the eye from that surface is constantly changing, how does the brain do it? How does it behave usefully.
Here is an example to show you that the light is ambiguous: we look around and we see no ambiguity unless manipulated. In fact all images are ambiguous as far as the eye is concerned. Here we have the same surface under two differently coloured light: a green light and red light. Imagine you are viewing this world through this window and the only information you have is the light that falls onto that window. You can see that the light coming from the same ‘blue’ surface is completely different depending on the colour of its illumination. Of course the only information the brain receives are these two dots, one purple and the other blue. Here – in this next slide – is another example of in which two physically identical projections on the eye, which are identical in every way, come from very different sources. Those are the projections, so maybe you can see the projection in the upper left corner which are arising from the orange cube in direct light viewed through a transparent surface, whereas the other projection is coming from the yellow cube, in shadow viewed through a differently coloured transparent surface. So these physically identical have completely different meanings. And the only way the visual brain can figure out what these meanings are is through its experience, its past interactions with the world. Put another way, the brain must shape what it sees according to this interaction with the environment.
To better explain what I mean by this, here we have two identical simultaneous brightness contrast illusions. The two central tiles are physically the same, but they look differently bright. Now, if I add information to the scene that makes the left illusion more consistent with the two tiles coming from two different sources: one a surface in light and the other a surface in shadow, they now appear much more different than they did before. The illusion on the right, which is place in an image that is consistent with the two surfaces being under the same light, now appear very similar to each other. Note that nothing has changed locally in either image. Only the global information has changed. Which, now, is the illusion?
Of course what an illusion is based on what the visual system is trying to see or do. The usual assumption is that the visual system is trying to see the world as it is. But if the visual system is trying to create a visual meaning – whether that meaning be accurate or inaccurate – then of these perceptions are useful, which means neither is actually an illusion.
Ok another example: 4 identical gray tiles on the right and seven on the left and seven on the right. We change the local information and now those gray tiles on the left look blue, whereas the 7 gray tiles on the right look yellow. This of course is a very powerful effect, because these two coloured perceptions are opposite ends of perceptual colored space. Remember, the tiles themselves have not changed. Only the local information has changed.
So what we see in these examples is that the human brain is continually defining reality. Humans are not an exception. All visual system face the same challenge, and Warren wanted me to show a couple of examples of such system … bumblebees. Here we have the Bee Matrix, where we can completely control the entire visual history of a single bee, and see how this change in the statistics of its experience determines their behavior. Here is a bee foraging from an artificial flower, which is getting sugar water when it lands on the correct colour. We can train these bees to use context to solve the same problem that the human brain must solve.
We can also instantiate this same challenge (of recognising surfaces under different colours of light) in artificial visual systems. Here is a virtual agent that hasn’t evolved, and so it doesn’t know what to do. Eventually she actually commits suicide. The next one is one, which has evolved to solve this problem over many generations through ‘natural’ selection has learned to recognize blue surfaces from red surfaces under different lights. After they evolve, we can then look into their mini-brains and see how they are doing it.
So, in conclusion, the ambiguity of visual images the basic challenge that the visual brain evolved to solve. In solving this problem, the brain must be relativistic … it can’t do absolutes; it involves mechanisms to find relationships. Thus, why context matters – at all levels from the social all the way down to the colors we see – is that it determines the historical meaning of an image. And it is this historical meaning that gets instantiated in the brain. We discussed yesterday the idea of liquid architecture. Brain brain is is fluid structure that doesn’t see the world as it is. Not only would this be useless, but it is impossible. Instead, what we see is a representation of our past interactions with the world; we see something that is useful to see. Colour makes that obvious; Color arising from the linear visual spectrum of light is warp by our brains into a three dimensional perceptual space. There is no isomorphic relationship there between the physical world and our perception of it.
It is not possible to fully understand the brain and how it perceives and creates if we abstract the brain from its history and ecology since it is that history and ecology that is physically represented in the brain.