[FreeSynapse]

At some point, it would be nice to receive a paper documenting sufficient comparative empirical analysis of HTMs with other competing ML algorithms, such as ANNs, SVMs, etc. Such an analysis should use reasonably optimal configuration of each model under comparison. It should cover multiple problem datasets, including some very high dimensional ones, and use various performance measures. Depending upon the resources available, meta algorithms may or may not be used. I look forward to seeing HTMs take the lead in this study, although I won't be disappointed if the initial version doesn't.

[Mir]

I have build my small-scale version of HTM, and I am in process of testing it on MNIST-Handwritten Digits set.

Since algorithms for sequence learning are not very obvious to build (to express it polite ), and I did not want to loose too much time on it, and since Mumenta release is coming, my version of HTM is just subset of the Numenta-release. My goal is also not to make my own HTM-Version, since I think it is pointless, becouse from everything I have read and heard, Numenta-people are doing good job. And also from the fairness reasons -- after all the theaory Hawkins and his collegues have done so far, it would be in my mind unfair to say: I can build better HTM.
The reason for my small version is simply because I could not wait for the release.

I'm still in process of doing some large-scale tests on MNIST (and for this data sets you have results published of all ML methods so far- inclusive NN, SVM, Convolution Nets, k-NN, linear classifiers etc), but my initial results look promising (I dont want to say more becouse I am not done yet).

It is not about that I expect 100% accuracy with HTM on this dataset (people have around 1.5%-2% error on the same data test and for some of the digits you have just to guess - so noisy). It is about testing all this old problems with ML-methods, e.g.

- if the model has a lot of 'degrees of freedom' you will overtrain it and get bad generalization (on unseen data).
- if it does not have enough 'degrees of freedom', you will get simplified model, which can not 'cover' all data
- how much training data it needs? In my view it is pointless to train the model with all 40.000 digits. The problem with all current ML methods on this data set (and all others) is: If you train it with just 10.000 digits (or less) you will have much worse results on test set.

This problems are, which I think is important to adress and compare with other methods in my testing of HTM, more than just 'performance on the test set' -- which is of course also important.

More on that, when I'm finished, and after I have access to 'real' HTM from Numenta. May be also on developer conference of Numenta, who knows .

Mir

[FreeSynapse]

[FreeSynapse]

The following is a question for Numenta:

Are there any predecided datasets in particular with which Numenta will experiment (or is experimenting) with the HTM framework? If so, I and/or others may want to run other ML learning algorithms on the same datasets. This will aid in comparing results. Thanks.