As you know from my last post, for the past 5–6 weeks, I have been undertaking (very) extensive data-science trials. The one specific problem on which I have been working, during this time, is: the MNIST dataset.
About the MNIST problem:
Essentially, the MNIST problem is to teach a computer how to recognize hand-written digits. There are tons of materials on this topic, google on “MNIST” to know more.
MNIST has been called the “hello world” program of machine learning. The idea is that MNIST is merely a toy problem with which you begin exploring the vast area of Artificial Intelligence.
MNIST also has been described as the “drosophilia” of machine learning. IMO, this second characterization is more accurate. After all, OCR of digits is one problem that has kept an entire generation of PhD students and researchers burning oil at night!
Progress over time in the prediction accuracy on the MNIST dataset:
The accuracy in recognizing the MNIST digits has gone up from the earlier levels like some 80 % 20 years ago to 90+ % about 10 years ago, and then on to 99 % being routine today.
As the accuracy kept on increasing, to make reports easier on the eye (i.e., to avoid too many 9’s), people started reporting errors rather than correct results. Thus, instead of saying “I got 99.52 % accurate result”, people these days routinely just say: “I got 0.48% error rate.”
For testing your algorithms/models, the standardized test data provided by the MNIST dataset consists of exactly 10,000 samples. This well-rounded figure turns out to be quite convenient, because when you achieve, say, a 99.65 % accuracy level, you not only are 0.35 % wrong, but the actual number of samples on which you went wrong also is precisely 35. So, it’s very easy to convert between absolute error rates and percentage ones.
To come back to the progress in accuracy, with today’s libraries like PyTorch, TensorFlow and Keras, it indeed seems that MNIST has become a toy problem. Even a simple ‘net search will get you to hundreds of tutorials which show you “how to get into top 1 %”. [Sometimes, this is just a piece of hype which means nothing more than: how to get 99.0 % accuracy—not how to get a 99-th percentile rank!
In other words, the trouble lies in the upper reaches of the accuracy scores. Each incremental progress from 99.65 % to 99.70 % to 99.75 % becomes harder and still harder—or even plain impossible to achieve, unless you basically improve your algorithm.
Despite throwing super-computers at it, no one has been able to achieve a 100.0% score on this dataset.
One reason for failing to achieve the 100% accuracy level is that some of the samples have come from people with astoundingly bad hand-writing.
Another reason for the failure is that a few of the standardized test samples (on which every one measures his accuracy) have in fact been wrongly labelled in the first place! For instance, the actual written-down digit may be “4”, but its standardized label says that it is “5” (because the guy who wrote that digit might have himself wrongly labelled it—or was a plain retard!). Now, once it’s standardized, if your algorithm actually guesses this sample right, then you lose on the accuracy as measured by the standardized data! This fact goes into the reason why it’s impossible to achieve 100.0% accuracy on the MNIST dataset.
Another trouble: There perhaps are too many ambiguous samples in this dataset. Two different algorithms may report the same accuracy (or error) level, say 26 errors. However, the particular samples on which they fail may not be exactly the same. In other words, there are many samples which show a tendency to get mis-classified on one algorithm but not on others!
Anyway, enough about these general remarks. Now, on to some hard data on the world-records.
The world’s best of the best:
While there are several Web pages that track the progress in the accuracy on the MNIST dataset, I found that many of the reliable ones have become outdated by now. In fact, today’s rank # 1 result does not appear in any of these compiled lists; I got to know of it through several Google searches.
So, here are the top 10 performances on the MNIST dataset, as found by me, as of today:
- Rank # 1: Number of Errors: 16. Accuracy: 99.84%
- Rank # 2: Number of Errors: 17. Accuracy: 99.83%
- Rank # 3: Number of Errors: 18. Accuracy: 99.82%
- Rank # 4: Number of Errors: 21. Accuracy: 99.79%
- Wan, Zeiler, Zhang, LeCun, and Fergus. (2013) [ PDF from LeCun’s site: ^]
- Rank # 5: Number of Errors: 23. Accuracy: 99.77%
- Cireşan, Meier, and Schmidhuber. (2012)
- Sato, Nishimura, and Yokoi. (2015)
- Rank # 6: Number of Errors: 24. Accuarcy: 99.76%
- Chang, and Chen. (2015)
- Bochinski, Senst, and Sikora. (2017)
- Deotte.(2020) Rank # 1 on the Kaggle Leaderboard on the MNIST dataset.
Noted on the Kaggle Leaderboard [^]. Comment: I suppose Deotte’s model carries over to the original MNIST dataset too, with at least this much accuracy. The two datasets differ. Kaggle has only 48 K of training data.
- Rank # 7: Number of Errors: 25. Accuracy: 99.75%
- Baldominos, Saez, and Isasi. (2018)
- Rank # 8: Number of Errors: 27. Accuracy: 99.73%
- Wan, Zeiler, Zhang, LeCun, and Fergus. (2013)
- Cireşan, Meier, Gambardella, and Schmidhuber. (2011)
- Rank # 9: Number of Errors: 28. Accuracy: 99.73%
- Baldominos, Saez, and Isasi. (2019)
- Rank # 10: Number of Errors: 28. Accuracy: 99.73%
- Lee, Gallagher, and Tu. (2016)
- Alom, Hasan, Yakopcic, and Taha. (2017)
Entries without explicit references to papers were obtained from the following source:
Alejandro Baldominos, Yago Saez, and Pedro Isasi, “A survey of handwritten character recognition with MNIST and EMNIST,” Appl. Sci. 2019, vol. 9, pp. 3169; doi:10.3390/app9153169 [^]
This source is more comprehensive and up to date (except for not including the entries for the top 3 ranks which came after the publication of the paper). Other sources, well-known but not always very up-to-date, are those by Prof. LeCun himself [^], Dr. Rodrigo Benenson [^], Benchmarks.AI [^], and the Wiki [^].
My current results (as of 13th April 2020, 11:00 IST):
Out of tens of trials conducted, let me pick out the two latest (and best) results which I’ve got so far:
- Trials group 1 (my private code: 03.05)
- Total training time: 23.3 hours (on my CPU-only machine)
- Number of Errors: 24. Accuracy: 99.76 %.
- Estimated world-rank: Together with the existing # 6.
- Trials group 2 (my private code: 03.06)
- Total training time: 27.5 hours (on my CPU-only machine)
- Number of Errors: 22. Accuracy: 99.78 %.
- Estimated world-rank: Above the existing rank # 5.
It must be noted that a significant reason for getting good accuracy levels is that I implement some new ideas of my own (parallels to which I have not seen anywhere else so far).
How do my results compare with the best in the world? Would it be possible to improve it?
Recall that I have a CPU-only machine. (I can’t buy a better machine because I don’t have any money, and in fact am in loans. I have been out of job for 2 years by now.)
I anticipate that if I were to have a GPU-enabled laptop, I should have crossed the 99.80 % accuracy level. This is not a vacuous statement; it comes with two different observation points:
Point No. 1:
Dr. Chris Deotte uses an ensemble of 15 models, each of which is run to 45 epochs. He gets to 99.75% accuracy. He is currently at # 1 position on the Kaggle leaderboard for MNIST.
Running Deotte’s one model with 45 epochs takes about 132 minutes on my machine. So, running an ensemble of 15 models would take about 1984 minutes, i.e., about 33 hours.
On the plus side, the Kaggle dataset has only 48 K training samples, and Deotte’s model still gets to 99.757 % accuracy (after averaging). It’s conceivable that with full 60 k training samples of the original MNIST, this architecture might go a few points higher.
On the minus side, when I ran Deotte’s architecture on my machine (as mentioned just above), I used the full 60 k of the original MNIST dataset. I still got only 99.70 % as the best accuracy. So, going much beyond 99.76 may not be possible, due to statistical nature of the training process.
To summarize this point:
33 hours on Deotte’s architecture might take you to 99.76 to 99.78 % accuracy, if you use the full 60 K training samples of the original MNIST dataset.
In comparison, my architecture (which makes use of Deotte’s insights) takes 24 to 28 hours, and definitely has gone up to 99.78 % accuracy already.
Point No. 2:
I have merely browsed through the architectures of the top 5 papers, not studied them. I presume that all of them use powerful parallel processing, at least GPUs, and that none uses a CPU-only machine (the way I do).
The top papers don’t usually give actual execution times required for training. However, one thing is striking.
The Rank # 1 paper uses a single model of about 1.5 million parameters, with which it goes up to 99.82 % (statistically best result). Using an ensemble of such models, it goes to 99.84 %.
In contrast, in my biggest architecture, each model of the ensemble was limited to fewer than half a million parameters (around 400 k).
The total number of parameters per model, taken by itself, is only an indicative measure, not conclusive. Details of the architecture matter, and so does the fine-tuning of a plethora of hyper-parameters. Processing differs. For instance, batch-normalization hardly adds any trainable parameters, but still results in a lot of extra training time (as well as some benefit for not overfitting or better accuracy).
To reach a conclusion on this point:
There is a definite margin to think that my new ideas might significantly enhance accuracy with even bigger architectures up to the tune of 1.5 million parameters.
Reasons for my present level of success, and future directions:
Let me highlight it again. I got (what to my mind are such) great results because of some new ideas that I thought of, and tried.
I mean to say, I tried everything “in the book,” and then some more. Thus, my model uses a deep network of CNNs, and ideas like batch normalization, dropout, data augmentation, and ensembling. That is to say, all the tricks that Dr. Chris Deotte explains so helpfully here [^] and puts to use here [^]. I used almost all of those tricks (or as many as I could, given my computing budget). And then, I added a few more tricks, based on my own ideas and thinking.
I have been abstractly thinking about some new ideas for image recognition for quite some time—for months, may be, and in one case, for almost 1.5 years or so. A group of closely related ideas from this brain-storming began acquiring a better shape late last year—though it was nothing immediately translatable into code. Then, I got some got to some fairly well concretized form in the first half of February 2020 (on the evening of the12th of that month, while in a restaurant, if what I remember is right).
However, in late February 2020, we had to make a move (i.e. shift everything to new house). A lot of time got “wasted” in it. So, I could start writing code only from 03 March 2020.
By 15th March 2020 (the date of my last post), it was already becoming increasingly clear to me that my ideas aren’t going to work out in their initial form. Yet, as I continued writing code and trying out improvements, these earlier ideas got transmutated into further new ideas. It is these latter ideas which were then implemented on bigger and various models, using TensorFlow and Keras.
There remain a few of these last group of ideas (the “transmutated” ones) which I do have in a fairly direct form, but which I could not actually try out so far, mainly because of the low computational power that I have. Trials take too much time, occupy the only machine that I have.
Trials are boring, in a way—while they are in progress. They are extremely boring, in fact, while they last. For hours and hours, you can do nothing but almost helplessly watch the stream of numbers appearing on the screen. You can’t even make out whether the idea you are trying is progressing for the better or the worse, because a handful of numbers at a time don’t give you any idea about the progress over several hours. Even if the current effort is actually worthless, you get to know only after hours, may be on the evening the next day. So, it’s all boring in the meanwhile, and tends to be draining in a way. That’s why, with 6 weeks spent like this, I am also nearing the fag-end of my patience.
Therefore, for now, I have decided to undertake just one more trial—a biggest one that I can still manage on my machine. I estimate that this single trial will go on for some 48 hours of run-time on my machine or more. Thanks to the TF-Keras, I can interrupt the trials once in a while, and resume training. So, it might take something like 4–5 days for this set of trial runs to get over.
I will see how this planned trial goes, and then will close this research on MNIST one way or the other. The reason is, I need to pursue other datasets like Fashion-MNIST, EMNIST, Not-MNIST, CIFAR-10, etc. too. My ideas need to be tested on those grounds too.
In the meanwhile, I also plan to update my resume soon enough, to include the current success on this research. (Yes, it more than amply qualifies for a full time engagement running for about 1.5 months!)
So, as far as MNIST is concerned, expect at the most one more blog post, if at all. In any case, for obvious reasons, I am going off the blog for another week.
One final clarification: I will not share the ideas or the code:
Take my report as being truthful or otherwise. I don’t care. I am not going to share the code. The ideas are patentable, and some of them are of more general nature. So, I am going to proceed very cautiously in sharing any details to any one. Sorry, but that’s how it is.
Credits and Acknowledgement:
While a whole range of materials by a lot of people helped me get here, let me single out three references/people over all others:
- Dr. Michael Nielsen, for his online book on ANNs [^]. He expediated my learning time by a few factors over what would have happened if I were to use those usual university text-books. I got most of my ANN theory from him. Some of the ideas I now tried here had struck me, at least in the seed form, while learning ANNs from this book in 2018. (It’s good that I never shared those ideas back then!)
- Dr. Jason Brownlee (again an Aussie!) [^], for his excellent online tutorials, and very simple and elegant toy code that helps get a feeling of having mastered some particular topic.
- Dr. Chris Deotte, for his most excellent code [^], and even more helpful to me, his article detailing how he got to his Kaggle rank # 1 result [^]. I might have come to know about the best existing practices on CNNs and MNIST by consulting the original papers. But Deotte cut the time for people like me by months, perhaps even a couple of years, via just those two articles at Kaggle.
Thanks to all the three of them! And, thanks also to hundreds of people on different fora (like StackExchange) sharing insights, comments, fastest running code snippets, etc.
To conclude this post:
A poor jobless Indian man has achieved a 99.78% accuracy score on the MNIST dataset using a CPUs-only machine. He thus has an estimated rank of # 5 in the world today. His result has an accuracy better than that of the top-ranking kernel for the MNIST dataset on the Google-sponsored Kaggle’s leaderboard. He has further ideas to enhance accuracy, which, to his mind, are promising. However, he is limited by the available computation power of his machine. But in any case, he is not going to share any of the ideas or code with any one else.
Addendum: Also, the same man hopes that at least now, with this world-class achievement, the Indian/Foreign IT companies/recruiters/HR/Top Management idiots would stop asking him to take their tests before proceeding with any employment consideration. If they can manage to get their “high-quality” people / geniuses with their tests and without reference to world-class achievements, well and good for them; he wishes them further “success.”
A song I like:
(Hindi) पीछे पीछे आ कर, छू लो हमें पा कर (“peechhe peechhe aa kar, chhoo lo hame paa kar”)
Singers: Lata Mangeshkar, Hemant Kumar
Music: S. D. Burman
Lyrics: Sahir Ludhianvi
[This song has a very fresh feel overall—just like so many songs that came from the Dev Anand + SD Burman combination. I don’t know why this song escaped me for such a long time while writing the songs section. (I’ve been listening to it off and on. It’s just that while actually writing the songs section, this song didn’t strike me, that’s all)
… As an exception, I can recommend the video too. I liked the picturization here, though the photography is techically not so sound (the contrast is not well managed). But it’s Mahabaleshwar—my favorite area. Kalpana Kartik’s dancing (jumping around, really speaking!) is so natural. Hardly anything of today’s will ever even approach this kind of an utter naturalness, this kind of a freshness. BTW, to those who don’t know, Kalpana Kartik and Dev Anand were husband and wife in real life. I guess they had already married (1954) by the time this movie was shot (released 1955). … So, this song is like 65 years old!…
Anyway, stay at home, wear mask (the droplet sizes are relatively big so masks can definitely help, and even the air-flow dynamics around a single virus particle, at that small a scale, should be almost Brownian, not of Navier-Stokes), wash your hands, and engage your mind in something else… Should be back after after a week or 10 days. Take care and bye for now…]