[Rob]

Today's Wall Street Journal article on Numenta mentioned that Dileep George had developed a mathematical model of Hawkins' theory. Does anyone know if this has been published somewhere? Unlike most of you, I don't think the ultimate answer lies in a software/microprocessor application. I think it will require reconfigurable hardware. I think a mathematical model will help in the development of such hardware.

[coulls]

Rob:

Actually he has a paper in submission to a neural networks conference. The paper can be found at this site: http://www.stanford.edu/~dil/invariance/. From my modest background in AI (very modest at that), I would say it isn't really possible to implement in hardware per se as it is basically a typical Bayesian belief network.

Hope this helps.

[NeuronExMachina]

As coulls mentioned, Dileep George has a research page here:

http://www.stanford.edu/~dil/invariance/

That page includes a PDF of the publication and some Matlab source code.

He organized a workshop on invariant representations in vision last weekend at Cosyne, one of the major computational neuroscience conferences. The list of abstracts is a pretty good read:

http://www.stanford.edu/~dil/cosyne05/index.html

Here's the relevant info on the paper:

http://www.stanford.edu/~dil/invariance/Download/GeorgeHawkinsIJCNN05.pdf

Title: A Hierarchical Bayesian Model of Invariant Pattern Recognition in the Visual Cortex

Dileep George and Jeff Hawkins, Stanford University and Redwood Neuroscience Institute
Accepted for publication in the proceedings of the International Joint Conference on Neural Networks. (IJCNN 05)

Abstract: We describe a hierarchical model of invariant visual pattern recognition in the visual cortex. In this model, the knowledge of how patterns change when objects move is learned and encapsulated in terms of high probability sequences at each level of the hierarchy. Configuration of object parts is captured by the patterns of coincident high probability sequences. This knowledge is then encoded in a highly efficient Bayesian Network structure. The learning algorithm uses a temporal stability criterion to discover object concepts and movement patterns. We show that the architecture and algorithms are biologically plausible. The large scale architecture of the system matches the large scale organization of the cortex and the micro-circuits derived from the local computations match the anatomical data on cortical circuits. The system exhibits invariance across a wide variety of transformations and is robust in the presence of noise. Moreover, the model also offers alternative explanations for various known cortical phenomena.

[DavidOlmsted]

Actually, I expected better.

Here we have another Bayesian Belief network (popular now for about the last 15 years), one that does not allow feedback loops (acyclic), and one that is used as a data structure only requiring a separate program be used to determine whether a pattern match exists. It also just characterizes static patterns in ways that have been done before. It breaks no new ground. In other words, it is not the killer app that the memory prediction model needs.

I also wish these scientific papers would stop playing games (I am not picking on this one only but all scientific papers about brain function) by only presenting what works instead of what doesn't work. This just reinforces my belief that progress in understanding the brain will only come from those working and publishing outside of the scientific establishment.
_________________
Click to go to my site at neurocomputing.org

[Rob]

David,
I think the AI community is too wed to old models. I think we still need to move towards a more distributed-intelligence hardware based model. Not that I have anything figured out, but someday...

[coulls]

I agree whole-heartedly with DavidOlmsted. I read this book and instantly I had some interesting ideas to think about and to build into my own 'intelligent machine' frameworks (more like ideas than frameworks, but who is counting?). Actually, when I read that Jeff Hawkins was creating Numenta with an implementation of ideas in this book, I was immediately discouraged because I figured that he took the ideas and created essentially what I had wanted to build. I was pleasantly suprised when I saw that in reality it is just a novel implementation of a Bayesian network.

It is my firm belief (though totally unprovable at this point) that strict probablity notations aren't the right way to model the connections made by neurons with their neighbors in 3-dimensional space. It occurs to me that the release and absorbtion of neurotransmitters just doesn't really mesh well with typical conditional probability. I also don't believe that some systems, such as Cyc, where knowledge bases are created a priori will be of any use. Have rules generated by humans doesn't make for intelligent or dynamic behavior in the least.

Anyway, I agree that these models won't lead to anything truly interesting or revolutionary. As a matter of fact, the stagnation and polarization of the AI field is one of the reasons that I never really got involved in AI research even though it is hugely interesting to me. The other reason is the shear number of papers that would have to be read in order to intelligently publish and create research directions.

I'd be interested in discussing some other ideas for the modeling of neuronal interactions, if anyone is interested. I am a grad student doing research in computer security areas, so right now I have no time to actually implement anything, but hopefully when summer comes around I can take a serious stab at some of my ideas.

On a side note, I'm actually quite happy I found this message board. It is often difficult to find an active community of people interested in AI that is open to discussing ideas with 'newbies', as they say.

[cl]

David, when you say it doesnt allow feedback loops, could you describe how a loop would operate in the context of this paper. It does allow feedback to inform lower level modules (not a loop i guess.) Looking at the results, (the graphs and what not,) if one were to scale this system up a lot and also make it 'predict forward in time' I think this could be revolutionary. It just seems like if one module is active while an object is present anywhere in the scene then your problems are solved. Surely this isnt all intelligence is, but it seems to be the root of it. I just havent heard of many invariant recognition systems. I am also fairly new to this whole deal so fill me in.

[coulls]

cl:

I believe what Dave meant was that patterns are not 'fed back' into the hierarchy anywhere to represent input over time. From what I gather from the paper, it seems to me that he 'feed back' you are referring to is simply reinforcement of probability values in the various tables of the hierarchy.

I do think this is a bit of an oversight, but it is probably beyond the scope of the work they published. After all, they were not using motion of the images or anything that is really 'time-based' to do their invariant recognition. What I am more disappointed in is the use of mathematical models that have been doing this stuff for a while. I agree that the hierarchical aspect of this is fairly interesting, but when it comes down to it you are simply doing conditional probabilities of different traits. What is the probability that this vertical line occurs with that vertical line at this time? Not exactly the stuff of revolution, and I would say not very faithful to the input signals that the brain handles.

I feel like there are other, perhaps more novel ways, of looking at the same sort of issues that move beyond conditional probabilities (perhaps investigating some set theoretic issues). If everything works out this summer, I will hopefully be spending some time coming up with such a system for use in intrusion detection (I am, after all, a computer security researcher). If the results seem promising versus known Bayesian approaches, then I will certainly be posting some of my early results and any publications here.

[b6s]

I can not agree with DavidOlmsted more.

I've just written a Chinese review about the book On Intelligence. Only one thing actually I would like to share with you guys here:

No matter how good an architecture of "memory" like HTM is, only human beings know (either consciously or unconsciously) the strategies to remember things or not. Statistical models such as BBP still need sophisticated algorithms, which are usually designed by graduate students (:p) to "calculate" probabilities of "chosen targets."

Just MHO.

[NuclearCat]

Not sure about whole Jeff theory, but at least in Dileep realization, I see two things not adequate:
1) Along layers we have only probabilities of some hypotesis travelling up and down, but I think in this scheme we need to formulate a context or this hypotesis itself into lower layers, for purpose of lower level should know which hypotesis upper layer actually means .
2) Number of such hypotesis is discrete, (and btw in generic belief propagation methods too), but in reality seems that no such discretization of hypotesis we have.