Next: Finite Growth Models Up: Topics in Computational Hidden Previous: Topics in Computational Hidden


Determining the computational nature of animate intelligence is perhaps the greatest single challenge facing the field of computer science. Many approaches are possible, and will no doubt be necessary if this riddle is to be solved. These range from the study of biological nervous systems, to analysis of the tasks that creatures perform - and frequently include a constructive component in which a machine is built or programmed to exhibit behavior that is in some sense intelligent. Here a distinction emerges. When the focus is entirely on successful performance of a particular task, one is led to consider designs that in all likelihood shed little light on the central question above. In its pure form this pursuit is then the study of artificial intelligence. That is, machines and software engaged in a masquerade - resembling in some respects the object of their emulation, but made of essentially different stuff. By contrast the study of synthetic intelligence strives towards constructions that while man-made, are nevertheless intelligent in the natural and animate sense. Of course in practice the distinction between synthetic and artificial is often hard to make, and really represents a difference in research direction and motivation. Moreover, while we may seem to attach a pejorative connotation to ``artificial'' in the discussion above, it certainly may be that machines will emerge from this work that are unquestionably intelligent in a deeply different sense from that with which we are today familiar. Nevertheless, our main interests lie along the synthetic direction.

Natural intelligence is impressively robust in the presence of noise and uncertainty. One view is that such factors should be dealt with by an outer wrapper of techniques, exposing an inherently discrete and symbolic problem at the core. Indeed there is a long history of investigators focusing on problems such as theorem proving and logical reasoning, while avoiding the less easily framed problem of robustness. Our view is that the manner in which nature deals with this more elusive problem may represent the central idea behind intelligence, not just a front-end noise filter.

Stochastic modeling techniques represent a formal approach to the problems of noise and uncertainty - and have been somewhat successful when applied to difficult problems such as speech recognition and other signal and image processing tasks.

Chapter 2 introduces the Finite Growth Model (FGM) framework which spans many existing model classes and opens up important new possibilities. Among these is the notion of stochastic transduction in which a machine converts one observation into another. The probability of transduction between two objects can be thought of as an indication of their similarity. A characteristic of natural intelligence is its use of nontrivial metrics, i.e. notions for similarity. Another striking feature is that these metrics are sometimes learned. Both of these are possible within the stochastic transduction paradigm. We remark that the well-known concept of string edit distance may be viewed as a single state memoryless transducer, and our work provides a convenient way to optimize its cost parameters. Speech recognition may be viewed as a grand transduction from signal to text. Chapter 3 describes experiments that represent a first step towards approaching the problem using this formalism. Finite growth models also include the class of hidden Markov models and stochastic context free grammars which can provide a means to discover hidden structure in a a set of observations. This corresponds to another salient characteristic of natural intelligence. FGMs also allow the model designer to cope with the learning-theoretic considerations of overtraining and generalization by building data-appropriate models resulting from optimization criteria such as that of minimum description length (MDL) or the maximum a posteriori probability estimate (MAP).

While interesting and perhaps practical, the stochastic modeling techniques we introduce would seem to be more artificial than synthetic - since a common language for this field is that of linear algebra, an unlikely component of our biological endowment. Beyond this observation, it seems unlikely that nature would confine herself to the use of strict probabalistic models.

An important contribution of chapter 2 is the presentation of FGMs, and by inclusion many specific stochastic model classes, in terms of weighted graphs and a related optimization problem. These constructions need not represent causal probability models - or probabilities at all. Nevertheless the celebrated Baum-Welch and EM algorithms are shown to still apply, and we argue that their essential message is one of decomposition. That is, breaking up a particular graph-based global optimization problem into a set of local problems such that progress on the local problems will necessarily advance the global objective. These observations paint FGMs in a far more connectionist light and it is for this reason that we see it as at least plausible that nature might employ related principles.

Chapter 4 sketches the design of a software library and language for FGMs. Experimentation in expedited by effective tools, and we argue that our design should be viewed as an assembly language for computational hidden-state stochastic modeling.

The material of chapters 5 and 6 is of a more esoteric and perhaps artificial nature. Chapter 5 exposes a fascinating and somewhat counterintuitive degeneracy in the relationship between the prior and a posteriori distributions arising from a mixture of normal densities. This degeneracy is then exploited to prove a reparameterization theorem that provides a modicum of theoretical justification to learning approaches that proceed by reweighting the input pattern set. In chapter 6 we discuss approaches to the learning of continuous context models.

We submit that a stochastic and information-theoretic paradigm for intelligence is emerging; although it is not clear whether the insights it generates pertain to the synthetic or only to the artificial face of the problem.

Next: Finite Growth Models Up: Topics in Computational Hidden Previous: Topics in Computational Hidden
Peter N. Yianilos