Are we living in the past?

There was an inspiring popular article published in Science Times (see this) with a long title "Are We Living In the Past? New Study Shows Brain Acts Like A Time Machine That Brings Us 15 Seconds Back". It caught my attention because the basic prediction of TGD inspired theory of consciousness is that the perceptive field is 4-dimensional rather than 3-D time=constant snapshot as in standard neuroscience.

The research article by Mauro Manassi and David Whitney (see this) with title " Illusion of visual stability through active perceptual serial dependence" suggests that visual perception is a kind of temporal average over a time interval, which can be even longer than 15 seconds.

1. The findings of Manassi and Whitney

1.1 Motivating question

"Why do the objects in the world appear to be so stable despite constant changes in their retinal images?" was the question that motivated the work of Manassi and Whitney. Retinal images continuously fluctuate because of sources of internal and external noise. Retinal image motion, occlusions and discontinuities, lighting changes, and perspective changes and many other sources of noise are present. However, the objects do not appear to jitter, fluctuate, or change identity from moment to moment. Why does the perceived world change smoothly over time although the real world does not?

This problem is also encountered in quantum consciousness theories. If conscious experience consists of a sequence of non-deterministic quantum jumps as moments of consciousness, it is not at all clear how a smooth stream of consciousness is possible.

One modern explanation for the smoothness of conscious experience is some kind of change blindness or inattentional blindness. The finite capacity of visual short-term memory is certainly a fact and forces a finite perceptive resolution and effectively eliminates too fast temporal gradients. This finite resolution poses limits in perceptual, decisional and memory processing. This would naturally apply also to other sensory memories.

In the standard view sensory percept corresponds to a time=constant snapshot of the physical world. The basic prediction is that the object at a given moment of time is the real object but in a finite perceptive resolution.

The alternative hypothesis studied in the article is that the visual system, and presumably also other sensory systems, use an active stabilization mechanism, which manifests as a serial dependence in perceptual judgments. Serial dependence causes objects at any moment to be misperceived as being more similar to those in the recent past. The serial dependence has been reported in the appearance of objects, perceptual decisions about objects, and the memories about objects. In all of these examples, serial dependence is found for random or unpredictable sequential images.

This raises the question whether one can understand the serial dependence by identifying the conscious perception at a given time as a weighted temporal average of preceding time= constant perceptions over some time interval T and what additional assumptions are needed to understand the other findings related to the phenomenon.

1.2 The experiments demonstrating the serial illusion

Article describes 5 experiments related to serial illusion. The experiments are described in detail in the article with illustrations (see this) and in the sequel I summarize them only very briefly. The reader is strongly encouraged to read the original article providing illustrations and references to literature related to serial illusion.

Experiment 1: shift of the perception to past

In Experiment 1 the shift of the perception to the past was demonstrated.

1. 2 separate groups of 44 and 45 participants rated the age of a young or old static face embedded in a blue frame (13 and 25.5 years, respectively). This gave a distribution of ratings around some mean identified as the real age of the face. The rating of the static face alone is referred to as the reference face .
2. A third group of 47 independent participants were presented with a movie of a face that morphed gradually, aging from young to old. These observers then rated the age of the old face. The rating of the static face preceded by the movie is referred to as the test face . The last frame of the video was identical to the reference face.
3. The age ratings between physically identical static faces, either alone (reference face) or with a preceding video (test face) were compared. Although the test and reference faces were identical, the old test face, seen after the video, was rated as 5 years younger than the old reference face, seen without the video (20.2 versus 25.5 years).
4. One can argue that the stability illusion is due to a simple unidirectional bias in age ratings. Therefore a fourth group of 45 new participants watched a movie of a face that gradually morphed from old to young. Following the movie, observers rated the age of a young static test face embedded in a blue frame. The young face was rated as 5 years older than its actual age (18.4 versus 13 years). Therefore the stability illusion can cause faces to appear younger or older depending on the previously seen faces.

These findings are consistent with the temporal averaging hypothesis.

Experiment 2: the effect of noise

The noise is known to increase the serial dependence. Whether this is the case also in the case of illusion stability was tested. Stimuli with and without noise were represented to separate groups of observers. As a measure of the stability illusion strength, attraction index as the bias in age ratings toward the beginning of the movie was introduced.

1. A measure of the stability illusion strength, attraction index was introduced. Attraction index is defined as Δ T/T , Δ T= | T_r-T_p| , where T_r is the real and T_p the perceived age of the test face, and T is the total age range T. Real age refers to the average perceived age in the Experiment without preceding video.
2. When the movie and test face were presented alone or with superimposed dynamic noise, the static test face ratings were attracted by 28 and 42 % of the movie.
3. When the movie was presented with increasing dynamic noise and a test face with high noise, the attraction was around 48 %.

   The results conform with the earlier finding that serial dependence in perception increases with noise and uncertainty. As the increasing dynamical noise yielded the strongest illusory effect, it was used across subsequent experiments.

Why should the increase of the noise increase the strength of the illusion stability? Suppose that the perception is average over time=constant perceptions from a time interval T. For instance, one could think of a Gaussian distribution for the weights of the contributions over the interval T. It would seem that T gets longer in the presence of noise in order to achieve reliability.

Experiment 3: Central tendency bias not involved

It might be argued that the results are due to a central tendency bias, i.e., the tendency to rate test faces as being close to middle age, independent of movie content.

To test this, Experiment 3 replicated the same conditions Experiment 1 but linear increase/decrease in the age of the face was replaced with a more complex increase/decrease using staircase functions leaving intact the starting and ending points of the movies (young and old).

1. Attraction index gradually decreased with decreasing the number of age steps in the movie, thus showing that our illusion is not only due to a simple response or central tendency bias but also strongly depends on the whole content of the face morphing movie
2. The attraction index was computed with the last 6, 18, and 30 seconds of the video preceding the test face. Attraction linearly increased with increasing video duration, thus showing that the attraction effect involves all parts of the preceding video.

These results seem to be consistent with the averaging hypothesis. If Gaussian distribution can be used to model the averaging, the parameter T characterizing the locus of the distribution was at least of order T= 30 seconds and that the distribution was rather flat in this range.

Experiment 4: Temporal strength/range of illusion

If our illusion is due to the proposed active mechanism of perceptual serial dependence, it should occur on a broad temporal range in accordance with previous findings.

In experiment 4 the temporal strength of the stability illusion with an interstimulus interval (I.S.I.) of 0, 1, 5, 10, and 15 seconds between the movie and test face was measured.

Test face age ratings were attracted toward the movie at all intervals, thus showing that stability illusion extends across a large period of time . These results also show that, without intervening trials, serial dependence magnitude extends over a larger period of time than previously shown.

Experiment 5: Face feature similarity

The previous serial dependence literature on face stimuli suggests that stability illusion should be determined by face feature similarity and should occur only when the face morphing movie and test face are similar (belong to the same person, and even more, have very nearly the same age).

Unlike previous passive change blindness based explanations, any modulation of the illusion respecting feature similarity should be consistent with serial dependence and would make it possible to make predictions about the perceived age T_p of the test face.

In experiment 5, a movie of a face that morphed from young to old was represented, and after an interval of 1 second, the age of the static test face was varied by making it younger or older than the original test old face. On the basis of the known tuning of serial dependence for face similarity, three predictions were formulated.

1. Stability illusion should occur only with faces similar in age to the test face and not between dissimilar faces. It was found that the old test face was rated as younger (attraction effect) only for a few similar identities that were most similar to the old face; the attraction disappeared for more dissimilar identities.
2. As the old test face was perceived as being ≈ 20 years old after watching the movie, it was predicted that, when a reference face that is 20 years old is used as a test face after the movie, the degree of attraction for that face should be zero. No attraction for a test face of 20 years of age was found.
3. Test faces younger than ≈ 20 years old should be perceived as older, because the movie content contains older identities across the duration of the morph movie and, hence, should bias test face perception toward older ages. When the test face was younger, it was rated as older than it actually was.

The results and predictions were very well captured by a two-parameter derivative of a Gaussian model, in accordance with previous results, and ideal observer models proposed in the serial dependence literature.

2. TGD based explanation for the findings

TGD inspired quantum theory of consciousness as a generalization of quantum measurement theory allowing to overcome its basic problem caused by the conflict between determinism of unitary time evolution and non-determinism of state function reduction (see for instance this). Zero energy ontology (ZEO) as an ontology of quantum theory (see this and this) plays a crucial role and leads to the proposal that the perceptive field is 4-dimensional so that one can speak of 4-D brain. This leads to a general vision about sensory perception and memory.

In the TGD framework, the question why the perceived world looks smooth is encountered already at quantum level. ZEO predicts two kinds of state function reductions (SFRs).

1. In "Big" SFRs (BSFRs) the arrow of time changes. In ZEO this explains in all scales why the world looks classical for the observer having arrow of time opposite to that for a system produced in BSFR (see this).
2. Sensory perceptions correspond naturally to "small" SFRs (SSFRs) and since SSFRs are the TGD counterparts of weak measurements of quantum optics and their sequences define what in the wave mechanics would correspond to a repetition of the same measurement (Zeno effect). Therefore one can hope that the problem disappears at quantum level.

   One must however understand why the perceived world seems to evolve smoothly although it does not.

The TGD based explanation for stability illusion and serial dependence relies on the general assumptions of TGD inspired theory of consciousness.

1. TGD inspired theory of consciousness predicts the notion of self hierarchy (see this). Self has subselves, which in turn have subselves which correspond to particular sub-subselves of self. Self experiences its subselves as separate mental images determined as averages of their subselves. There are therefore three levels involved: self, subself, and sub-sub-self. Self hierarchy is universal and appears in all scales and one can ask whether the super-ego--ego--Id triple of Freud could be interpreted in terms of this hierarchy.

   The correspondences are therefore "We" ↔ self; mental image ↔ subself; subself as mental images of self ↔ average over sub-subselves.

2. In accordance with the vision of the 4-D brain, ZEO makes possible the temporal ensemble of mental images as a basic element of quantum consciousness. No separate neural mechanism for forming the temporal ensemble is needed: its generation is a basic aspect of the quantum world.
3. The perception (subself) as a mental image is identified as a kind of temporal average over time=constant perceptions (sub-subselves), which basically correspond to quantum measurements and can in ZEO be identified as "small" state function reductions (SSFRs) in ZEO. Continuous stream of consciousness would replace the Zeno effect.

   The averaging smooths out various fluctuations (to which also SSFRs contribute at quantum level) and subselves as temporal averages over sub-subselves give rise to an experience of a smoothly changing world. The conscious sensory perception at "our" level is not about time=constant snapshot but an average over this kind of snapshots.

Consider now a model for the stability illusion and various aspects of serial dependence. In the following T_r resp. T_p denotes the real resp. perceived age (after seeing the video) of the face. T denotes the total age range. t_k denotes the time associated with k^th video picture and t_max the total duration of the video.

1. Sub-subselves in the experiments of Manassi and Whitney correspond to t=t_k<t_max video snapshots. Subself at t=t_k corresponds to a statistical average M_k of 0< r< k video snapshots at t_r. At t= t_k, "we" experiences M_k . The averaging over time gives rise to experience, which is biased towards earlier perceptions. The averaging creates the smoothing of the perception and generates the illusion that the perceived mental image is shifted to the past.

   If the perceived ages T_p,k, to be distinguished from t_k corresponding to real ages T_r,k=T₀+ kΔ T_r contribute with the same weight in the age interval T, the average corresponds to the central value of T=T₀+T/2. In the general case, the average depends on the details of the distribution for T_r,k and on the distribution of weights for t_k in accordance with the results of Experiment 3.

2. The higher the noise level, the longer the maximal time interval t_M over which the averaging takes place in order to gain reliability. This requires active response by changing t_M for M_k. t_M must increase with the noise level. For instance, if the weights in the average are Gaussian, the width of the Gaussian distribution must increase with the noise level. This explains the findings of Experiment 2 relating to the effects of noise.

Experiment 5 provides the information needed to formulate a model for what could happen in the addition of a new face at t=t_N.

1. The test face F_N+1 is first experienced as a different person. After that it is checked whether F_N+1 corresponds to any memory mental image M_k in the set {M_k| k = 1,...,N}. This involves memory recall besides time=constant snapshot perception.

   If F_N+1 is similar to some M_k in the set {M_k, k=1,..N}, it is added to M_N and defines a new memory mental image M_N+1 and there is a stability illusion.

   If it does not correspond to any M_k, it is not recognized as an already perceived face, and is not added to M_N as a new memory M_N+1 so that there is no stability illusion.

2. This model explains the results of 3 sub-experiments of Experiment 5 relating to the face feature similarity. The second experiment however deserves a detailed comment since it involves criticality in the sense that a small variation of the real age of F(N+1) should lead to a disappearance of the stability illusion.

   Let T_p,A be the perceived age of the test face in experiment A and T_r,B the real age in the next experiment. For T_B,r= T_A,p the stability illusion is absent whereas for T_B,r< T_A,p it is present. The situation is therefore critical.

   The proposed model explains the presence of the illusion. One can however argue that T_B,r> T_A,p rather than T_B,r= T_A,p should actually hold true, or more precisely, there was no memory mental image M_k with T_p<q T_r. A small variation of T_B,r makes it possible to test whether the situation is really critical.

See the article Quantum Statistical Brain or the chapter with the same title.

For a summary of earlier postings see Latest progress in TGD.

Articles and other material related to TGD.