[intelligent robot]

I believe that quite a few people on this forum (including myself) are working on their own implementations of the theories in On Intelligence.

I would like to ask everybody who is doing so to raise their hand and exchange ideas. There is a lot to be learned; and even Numenta could use some help

Tell us what you have in mind, what you have done, what programming languages you have used, your results, etc.

[keastm]

I am working on an Oracle based / PLSQL implementation of my understanding of what Jeff Hawkins describes as an HTM. I choose a database implementation for the training and persistence modules. In hindsight this was a good choice due to the large number of patterns that I'm storing during training.
I have a subject (cause) that I have coded 64 (data) input sensors for. Of the 64 inputs 62 are static and represent the cause throughout the temporal input phase. The 62 inputs are aggregated into 23 pattern groups in the first level of the hierarchy (75 pattern groups total). All told I have 7 levels in the hierachy.
There are 2 temporal inputs that are mapped into a single pattern group.
The single temporal pattern input is combined with 13 high level static input patterns (from various levels in the hierachy).
I am currently letting the prototye learn from real world data. I have no influence over (or idea) what it's learning.
My next development phase is to develop an inference and prediction module. I am in the eary development stages but don't see and major complications.
Of course, I really won't know if any of this is of any value until I get the prediction module into test. Anybody can build a pattern machine!

[joel]

I'm not really building an HTM model, but for the sake of keeping the conversation going, here's a bit about my project.

I've been interested in the effects of putting feature-filters between raw input patterns and associative arrays. Sort of a Kononen thing. I'm using the traditional input-summing binary threshold units. I've experimented with input patterns having various correlation characteristics (e.g purely random, line segments, textures), and various feature-filter rules to find that the storage capacity and recall radius of the associative array can be significantly increased. There's endless tweaks to the net and learning-rules to be looked at for further improvement of a generally useless system. Sigh...

My project is programmed in C/lex/yacc/make. As I'm working with relatively large nets, with hundreds to thousands of units, I wanted an efficient programming language. The simulator works along the lines of the old Rochester Connectionist Simulator (RCS). A unit array, each unit having a linked list of sites, each site having a linked list of links to other units. Each unit, site, and link can be assigned a function at network build time.

To take the thing in the HTM direction, I'd replace the binary units with real-valued units so beliefs could be propogated. Time can be built into the net in several ways. The original RCS method would be to put FIFOs in the units, sites, or links (units being cheapest computationally). Alternatively, an RC-like exponential (or any other function) could be put between the unit's input potential value and its output function.

In the longer run, I'd like to set up my network with each unit being an automaton. I think the statistics would play out better that way, and the approach would lend itself to hardware implementation.

All good fun! Let's hear from the rest of you!

[intelligent robot]

Thanks for the input, people.

I myself am working on simulations of neuron-like 'automata' that classify input patterns and sequences. Recently I executed one on an image that had had it's edges extracted and got encouraging results.
My system is exceedingly simple to understand and program, makes NO use of complicated math, and is highly adaptable and dynamic - just like real neurons. I can give the details if anyone is interested. The only drawback is that it is a little computationally expensive. It's nothing that can't be fixed, though.

I'm programming it in MinGW & FreeBasic.

I also have a theory of how consciousness may arise in a hierarchical system. My theory explains how a hierarchical organisation of unintelligent agents (anything from the neocortex to HTMs) may develop a united self-awareness. It also goes into explaining the conscious and unconscious minds.
It's still just a theory, though.

So, let's have some thoughts!

[DavidOlmsted]

I am also working on something else since my interest is in how the whole brain works and works together as a complete autonomous system. I believe one needs to start out with simple brains and work upward towards complexity. I am also hopeful such realistic simulations will eventually shed light on mental illness in addition to being a starting point for new technologies tailored for specific purposes.

You can see the latest at neurocomputing.org.
_________________
Click to go to my site at neurocomputing.org

[TaleStuff]

Hi,
I've written a simple Memory Prediction framework app for Palm OS
http://home.earthlink.net/~gmayhak/M5_htm.htm

Could this simulation be showing any sign of intelligence?

Gary

[Lawrence Phillia]

Although im not using jeff's version of an HTM in my own modelling, prediction does play a role in advanced control theory as it as for quite some time.

http://lorien.ncl.ac.uk/ming/advcontrl/sect1.htm

My projects are electronics and hardware intensive where, apart from an occational binary dump from an host computer, software is a small but vital part.

[mikehahn]

I admit to knowing next to nothing about neural networks and AI, but I just thought I'd add my two cents worth.

I would like to create a simplified computer model of the neocortex, which I call a Toody Machine. Toody inhabits a 2D, computer-generated world, can see an array of pixels, can pan up/down/left/right, and can click on any of the pixels she sees. She can also hear/speak one ASCII character at a time. To learn more about Toody, please visit:

http://www.treenimation.net/toody.html

[iammisc]

I haven't started anything yet, but I would like to get ideas for an implementation that I would really like to start soon. First of all I would like to know how all of you are implementing learning? How do you recognize patterns? Are your patterns absolute or do they account for minor differences in input? Do you explicitly train the network or do you let it learn by itself? How do you implement feedback? How do you represent names?

I was thinking of a more neuron-like approach. Basically, each neuron would look at the activity state of its under-neurons. The activity of each neuron is a function of the states of the neurons underneath the neuron and the expected state. Feedback is implemented by having the output of a parent neuron connected to each child neuron. I think this is something along the lines of what the brain does. However, the thing that confuses me is the fact that each neuron only spikes at one sequence of patterns. There is no name. What confuses me about my approach is that Hawkins says that there are name cells. But how do these name cells come to represent these names and a specific pattern? Won't all the name cells in V2 for example represent the same thing because the first pattern that they learned was the pattern of the state of V1. I don't think I'm making much sense right now. But here is a simple question whose answer, I believe, will shed some light on the problem that I am having: What piece of data does each node send to higher-level nodes? Hawkins says that it is either a name or an unexpected pattern. But how do you encapsulate the entire pattern formed by a neuron's inputs into only one output?? How is it possible that one connection could possibly encapsulate the state of a neuron's inputs?

I think that somewhere down the line, I misunderstood Hawkin's theories and I would appreciate it if you could clear it up for me.

Anyway, in a sort of a weird coincidence, while I was reading On Intelligence, I was already working on a distributed programming language that as of right now has pretty decent performance and can run large computations on about a 5 computer cluster. The programming language is based around the concept of actors, which are independent entities that consist of data and threads of execution. This concept lends itself perfectly to the idea of HTMS. Each actor could be one node in the Hierarchical temporal memory network and each actor would executed independently, exactly how the real brain works. I really want to get started on my own implementation of HTMs and I would appreciate it if you would steer me in the right direction.

Thanks again.

[Lloyd]

This is more a reply to iammisc than anything else, but it is somewhat relevant to the thread. I am not working on a specific simulation at this time, but I am working with a traditional neural net classifier and hope to add some other approaches, such as SVMs. I want to learn more about Bayesian math.

As I understand it, the work on "granny" neurons suggests that "names" are coded as firing patterns among specific groups of neurons. I imagine a sort of binary code in the sense of who is firing now and who is not among the group. I suspect that such groups might actually be flexible in constituency, rather than comprising a specific fixed set of neurons. That makes the math a lot messier, but evolution doesn't care about that.

Conceptually, I relate such groups to Gerald Edelman's work on what he calls reentrant systems. He did some early explorations relating these to the output of classifiers, such as neural nets. As I understand it, a reentrant system is a bundle of fibers which convey specific messages from cortex to elsewhere, primarily the thalamus. Or maybe the reentrant system includes a return tract, but either way, it's conceptually close, I think.

Landauer found that it took a rank of about 300 to classify millions of documents, using his SVD-based Latent Semantics system. And I believe that Edelman was finding similar numbers in his early work on robot guidance. So is there evidence for thalamo-cortical tracts of 300 axons, more or less? And it seems that an HTM approach would require similar tracts spreading between regions throughout the cortex. This would not surprise me, but I know of little evidence.