Sound separation, voice separation from a song, the cocktail party effect, etc., and AI/ML

A Special note for the Potential Employers from the Data Science field:

Recently, in April 2020, I achieved a World Rank # 5 on the MNIST problem. The initial announcement can be found here [^], and a further status update, here [^].

All my data science-related posts can always be found here [^].

1. The “Revival” series of “Sa Re Ga Ma” music company:

It all began with the two versions of the song which I ran the last time (in the usual songs section). The original song (or so I suppose) is here [^]. The “Revival” edition of the same song is here [^]. For this particular song, I happened to like the “Revival” version just so slightly better. It seemed to “fill” the tonal landscape of this song better—without there being too much of degradation to the vocals, which almost always happens in the “Revival” series. I listened to both these versions back-to-back, a couple of times. Which set me thinking.

I thought that, perhaps, by further developing the existing AI techniques and using them together with some kind of advanced features for manual editing, it should be possible to achieve the same goals that the makers of “Revival” were aiming at, but too often fell too short of.

So, I did a bit of Google search, and found discussions like this one at HiFiVision [^], and this one at [^]. Go through the comments; they are lively!

On the AI side, I found this Q&A at Quora: [^]. One of the comments said: “Just Google this for searching `software to clarify and improve sound recordings’ and you will have several thousand listings for software that does this job.”

Aha! So there already were at least a few software to do the job?

A few more searches later, I landed at Spleeter by Deezer [^]. (In retrospect, this seems inevitable.)

2. AI software for processing songs: Spleeter:

Spleeter uses Python and TensorFlow, and the installation instructions assume Conda. So, I immediately tried to download it, but saw my Conda getting into trouble “solving” for the “environment”. Tch. Too bad! I stopped the Conda process, closed the terminal window, and recalled the sage advice: “If everything fails, refer to the manual.” … Back to the GitHub page on Deezer, and it’s only now that I find that they still use TF 1.x! Bad luck!

Theoretically, I could have run Spleeter on Google Colab. But I don’t like Google Colab. Nothing basically wrong with it, except that (i) personally, I think that Jupyter is a great demo tool but a very poor development tool, and (ii) I like my personal development projects unclouded. (I will happily work on a cloud-based project if someone pays me to do it. For that matter, if someone is going to pay me and then also advices me to set up the swap space for my local Ubuntu installation on the cloud, I will happily do it too! But when it comes to my own personal projects, don’t expect me to work on the cloud.)

So, tinkering around with Spleeter was, unfortunately, not possible. But I noticed what they add at the Spleeter’s page: “there are multiple forks exposing spleeter through either a Guided User Interface (GUI) or a standalone free or paying website. Please note that we do not host, maintain or directly support any of these initiatives.” Good enough for me. I mean, the first part…

So, after a couple of searches, I landed at a few sites, and actually tried two of them: [^] and [^]. The song I tried was the original version of the same song I ran the last time (links to YouTube given above). I went for just two stems in both cases—vocals and the rest.

Ummm…. Not quite up to what I had naively expected. Neither software did too well, IMO. Not with this song.

3. Peculiarity of the Indian music:

The thing is this: Indian music—rather, Indian film (and similar) music—tends to be very Indian in its tastes.

We Indians love curries—everything imaginable thrown together. No 5 or 7 or 9 or 11 course dinners for us. All our courses are packed into a big simultaneous serving called “thaali”. Curries are a very convenient device in accomplishing this goal of serving the whole kitchen together. (BTW, talking of Western dinners, it seems to me that they always have an odd number of courses in dinners. Why so? why not an even number of courses? Is it something like that 13 thing?)

Another thing. Traditionally, Indian music has made use of many innovations, but ideas of orchestra and harmony have never been there. Also, Indian music tends to emphasize continuity of sounds, whereas Western music seems to come in a “choppy” sort of style. In Western music, there are clear-cut demarcations at every level, starting from very neatly splitting a symphony into movements, right down to individual phrases and the smallest pieces of variations. Contrast it all with the utter fluidity and flexibility with which Indian music (especially the vocal classical type) gets rendered.

On both counts, Indian music comes across as a very fluid and continuous kind of a thing: a curry, really speaking.

All individual spices (“masaalaa” items) are not just thrown together, they are pounded and ground together into a homogenous ball first. That’s why, given a good home-made curry, it is tough to even take a good guess at exactly what all spices might have gone into making it. …Yes, even sub-regional variations are hard to figure out, even for expert noses. Just ask a lady from North Maharashtra or Marathwada to name all the spices used in a home-made curry from the north Konkan area of Maharashtra state (which is different from the Malavani sub-region of Konkan), going just by the taste. These days, communications have improved radically, and so, people know the ingredients already. But when I was young, women of my mother’s age would typically fail to guess the ingredients right. So, curries come with everything utterly mixed-up together. The whole here is not just greater than the sum of its parts; the whole here is one whose parts cannot even be teased out all that easily.

In India, our songs too are similar to curries. Even when we use Western instruments and orchestration ideas, the usage patterns still tend to become curry-like. Which means, Indian songs are not at all suitable for automatic/mechanically conducted analysis!

That’s why, the results given by the two services were not very surprising. So, I let it go at that.

But the topic wasn’t prepared to let me go so easily. It kept tugging at my mind.

4. Further searches:

Today, I gave in, and did some further searches. I found Spleeter’s demo video [^]. Of course, there could be other, better demo’s too. But that’s what I found and pursued. I also found a test of Spleeter done on Beatles’ song: “Help” [^].

Finally, I also found this video which explains how to remove vocals from a song using Audacity[^]. Skip it if you wish, but it was this video which mentioned Melodyne [^], which was a new thing to me. Audacity is Open Source, whereas Melodyne is a commercial product.

Further searches later, I also found this video (skip it if you don’t find all these details interesting), using ISSE [^]. Finally, I found this one (and don’t skip it—there’s a wealth of information in it): [^]. Among many things, it also mentions AutoTune [^], a commercial product. Google search suggested AutoTalent as its Open Source alternative; it was written by an MIT prof [^]. I didn’t pursue it a lot, because my focus was on vocals-extraction rather than vocals pitchcorrection.

Soooooo, where does that leave us?

Without getting into all the details, let me just state a few conclusions that I’ve reached…

5. My conclusions as of today:

5.1. Spleeter and similar AI/ML-based techniques need to improve a lot. Directly offering voice-separation services is not likely to take the world by the storm.

5.2. Actually, my position is better stated by saying this: I think that directly deploying AI/ML in the way it is being deployed, isn’t going to work out—at all. Just throwing tera-bytes of data at the problem isn’t going to solve it. Not because the current ML techniques aren’t very capable, but because music is far too complex. People are aiming for too high-hanging a fruit here.

5.3. At the same time, I also think that in a little more distant future, say over a 5–10 years’ horizon, chances are pretty good that tasks like separating the voice from the instrumental sounds would become acceptably good. Provided, they pursue the right clues.

6. How the development in this field should progress (IMO):

In this context, IMO, a good clue is this: First of all, don’t start with AI/ML, and don’t pursue automation. Instead, start with a very good idea of what problem(s) are at all to be solved.

In the present context, it means: Try to understand why products like Melodyne and AutoTune have at all been successful—despite there being “so little automation” in them.

My answer: Precisely because these software have given so much latitude to the user.

It’s only after understanding the problem to be solved, and the modalities of current software, that we should come to this question of whether the existing capabilities of these software can at all be enhanced using AI/ML, using one feature/aspect at a time.

My answer, in short: Yes, they can (and should) be.

Notice, we don’t start with the AI/ML algorithms and then try to find applications for them. We start with some pieces of good software that have already created certain needs (or expanded them), and are fulfilling them already. Only then do we think of enhancing it—with AI/ML being just a tool. Yes, it’s an enabling technology. But in the end, it’s just a tool to improve other software.

Then, as the next step, consolidate all possible AI-related gains first—doing just enhancements, really speaking. Only then proceed further. In particular, don’t try to automate everything right from the beginning.

IMO, AI/ML techniques simply aren’t so well developed that they can effectively tackle problems involving relatively greater conceptual scope, such that wide-ranging generalizations get involved in complex ways, in performing a task. AI/ML techniques can, and indeed do, excel—and even outperform people—but only in those tasks that are very narrowly defined—tasks like identifying handwritten digits, or detecting few cancerous cells from among hundreds of healthy cells using details of morphology—without ever getting fatigued. Etc.

Sound isolation is not a task very well suited to these algorithms. Not at this stage of development of AI/ML, and the sound-related softwares.

“The human element will always be there,” people love to repeat in the context of AI.

Yes, I am an engineer and I am very much into AI/ML. But when it comes to tasks like sound separation and all, my point is stronger than what people say. IMO, it would be actually stupid to throw away the manual editing aspects.

Human ear is too sensitive an instrument; it takes almost nothing (and almost no time) for most of us to figure out when some sound processing goes bad, or when a reverse-FFT’ed sound begins to feel too shrill at times, or too “hollow” at other times, or plain “helium-throat”-like at still others [^][^].

Gather some 5–10 people in a room and play some songs on a stereo system that is equipped with a good graphic equalizer. If there is no designated DJ, what is the bet that people are just going to fiddle around with the graphic equalizer every time a new song begins? The reason is not just that people want to impress others with their expertize. The fact of the matter is, people are sensitive to even minutest variations that come in sound, and they will simply not accept something which does not sound “just right.” Further, there are individual tastes too—as to what precisely is “just right”. That’s why, if one guy increases the bass, someone else is bound to get closer to the graphic equalizer to set it right! It’s going to happen.

That’s why, it’s crucial not to even just attempt to “minimize” the human “interference.” Don’t do that. Not for software like these.

Instead, the aim should be to keep that human element right at the forefront, despite using AI/ML.

Ditto, for other similarly complex tasks / domains, like colouring B&W images, generating meaningful passages of text, etc.

That’s what I think. As of today.

7. Guess I also have some ideas for processing of music:

So, yes, I am not at all for directly starting training Deep Learning models with lots of music tracks.

At the same time, I guess, I also have already started generating quite a few ideas regarding these topics: analysis of music, sound separation, which ML technique might work out well and in what respect (for these tasks), what kind of abstract information to make available to the human “operator” and in what form/presentation, etc. …

…You see, some 15+ years ago, I had actually built a small product called “ToneBrush.” It offered real-time visualizations of music using windowed FFT and further processing (something like spectrogram and the visualizations which by now have become standard in all media players like VLC etc.). My product didn’t even sell just a single copy! But it was a valuable experience…

…Cutting back to the present, all the thinking which I did back then, now came back to once again. … All the same, for the time being, I’m just noting these ideas in my notebook, and otherwise moving this whole topic on to the back-burner. I first want to finish my ongoing work on QM, first of all.

One final note, an after-thought, actually: I didn’t say anything about the cocktail party effect. Well, if you don’t know what the effect means, start with the Wiki [^]. As to its relevance: I remember seeing some work (I think it was mentioned at Google’s blog) which tried to separate out each speaker’s voice from the mixed up signals coming from, say, round-table discussion kind of scenarios. However, I am unable to locate it right now. So, let me leave it as “homework” for you!

8. On the QM front:

As to my progress on the QM side—or lack of it: I spotted (and even recalled from memory) quite a few more conceptual issues, and have been working through them. The schedule might get affected a bit, but not a lot. Instead of 3–4 weeks which I announced 1–2 weeks ago, these additional items add a further couple of weeks or so, but not more. So, instead of August-end, I might be ready by mid-September. Overall, I am quite happy with the way things are progressing in this respect. However, I’ve realized that this work is not like programming. If I work for more than just 7–8 hours a day, then I still get exhausted. When it’s programming and not just notes/theory-building, then I can easily go past 12 hours a day, consistently, for a lot longer period (like weeks at a time). So, things are moving more slowly, but quite definitely, and I am happy with the progress so far. Let’s see.

In the meanwhile, of course, thoughts on topics like coloring of B&W pics or sound separation also pass by, and I note them.

OK then, enough is enough. See you after 10–15 days. In the meanwhile, take care, and bye for now…

A song I like:

(Western, Pop) “Rhiannon”
Band: Fleetwood Mac

[I mean this version: [^]. This song has a pretty good melody, and Stevie Nicks’s voice, even if it’s not too smooth and mellifluous, has a certain charm to it, a certain “femininity” as it were. But what I like the most about this song is, once again, its sound-scape taken as a whole. Like many songs of its era, this song too carries a right level of richness in its tonal land-scape—neither too rich nor too sparse/rarefied, but just right. …If I recall it right, surprisingly, the first time I heard this song was not in the COEP hostels, but in the IIT Madras hostels.]

— 2020.08.09 02:43 IST: First published
— 2020.08.09 20:33 IST: Fixed the wrong linkings and the broken links, and added reference to AutoTalent.


Status update on my trials for the MNIST dataset

This post is going to be brief, relatively speaking.

1. My further trials for the MNIST dataset :

You know by now, from my last post about a month ago [^], that I had achieved a World Rank # 5 on the MNIST dataset (with 99.78 % accuracy), and that too, using a relatively slow machine (single CPU-only laptop).

At that time, as mentioned in that post, I also had some high hopes of bettering the result (with definite pointers as to why the results should get better).

Since then, I’ve conducted a lot of trials. Both the machine and I learnt a lot. However, during this second bout of trials, I came to learn much, much more than the machine did!

But natural! With a great result already behind me, my focus during the last month naturally shifted to better understanding the why’s and the how’s of it, rather than sheer chasing a further improvement in accuracy, by hook or crook.

So, I deliberately reduced my computational budget from 30+ hours per trial to 12 hours at the most. [Note again, my CPU-only hardware runs about 4–8 times slower, perhaps even 10+ times slower, as compared to the GPU-carrying machines.]

Within this reduced computing budget, I pursued a lot many different ideas and architectures, with some elements being well known to people already, and some elements having been newly invented by me.

The ideas I combined include: batch normalization, learnable pooling layers, models built with functional API (permitting complex entwinements of data streams, not just sequential), custom-written layers (not much, but I did try a bit), custom-written learning-rate scheduler, custom-written call-back functions to monitor progress at batch- and epoch-ends, custom-written class for faster loading of augmented data (TF-Keras itself spins out a special thread for this purpose), apart from custom logging, custom-written early stopping, custom-written ensembles, etc. … (If you are a programmer, you would respect me!)

But what was going to be the result? Sometimes there was some faint hope for some improvement in accuracy; most times, a relatively low accuracy was anyway expected, and that’s what I saw. (No surprises there—the computing budget, to begin with, had to kept small.)

Sometime during these investigations into the architectures and algorithms, I did initiate a few long trials (in the range of some 30–40 hours of training time). I took only one of these trials to completion. (I interrupted and ended all the other long trials more or less arbitrarily, after running them for, may be, 2 to 8 hours. A few of them seemed leering towards eventual over-fitting; for others, I simply would lose the patience!)

During the one long trial which I did run to completion, I did achieve a slight improvement in the accuracy. I did go up to 99.79 % accuracy.

However, I cannot be very highly confident about the consistency of this result. The algorithms are statistical in nature, and a slight degradation (or even a slight improvement) from 99.79 % is what is to be expected.

I do have all the data related to this 99.79 % accuracy result saved with me. (I have saved not just the models, the code, and the outputs, but also all the intermediate data, including the output produced on the terminal by the TensorFlow-Keras library during the training phase. The outputs of the custom-written log also have been saved diligently. And of course, I was careful enough to seed at least the most important three random generators—one each in TF, numpy and basic python.)

Coming back to the statistical nature of training, please note that my new approach does tend to yield statistically much more robust results (with much less statistical fluctuations, as is evident from the logs and all, and as also is only to be expected from the “theory” behind my new approach(es)).

But still, since I was able to conduct only one full trial with this highest-accuracy architecture, I hesitate to make any statement that might be mis-interpreted. That’s why, I have decided not to claim the 99.79 % result. I will mention the achievement in informal communications, even on blog posts (the way I am doing right now). But I am not going to put it on my resume. (One should promise less, and then deliver more. That’s what I believe in.)

If I were to claim this result, it would also improve my World Rank by one. Thus, I would then get to World Rank # 4.

Still, I am leaving the actual making of this claim to some other day. Checking the repeatability will take too much time with my too-slow-for-the-purposes machine, and I need to focus on other areas of data science too. I find that I have begun falling behind on them. (With a powerful GPU-based machine, both would have been possible—MNIST and the rest of data science. But for now, I have to prioritize.)


Since my last post, I learnt a lot about image recognition, classification, deep learning, and all. I also coded some of the most advanced ideas in deep learning for image processing that can at all be implemented with today’s best technology—and then, a few more of my own. Informally, I can now say that now I am at World Rank # 4. However, for the reasons given above, I am not going to make the claim for the improvement, as of today. So, my official rank on the MNIST dataset remains at 5.


2. Miscellaneous:

I have closed this entire enterprise of the MNIST trials for now. With my machine, I am happy to settle at the World Rank # 5 (as claimed, and # 4, informally and actually).

I might now explore deep learning for radiology (e.g. detection of abnormalities in chest X-rays or cancers), for just a bit.

However, it seems that I have been getting stuck into this local minimum of image recognition for too long. (In the absence of a gainful employment in this area, despite my world-class result, it still is just a local minimum.) So, to correct my overall tilt in the pursuit of the topics, for the time being, I am going to keep image processing relatively on the back-burner.

I have already started exploring time-series analysis for stock-markets. I would also be looking into deep learning from text data, esp. NLP. I have not thought a lot about it, and now I need to effect the correction.

… Should be back after a few weeks.

In the meanwhile, if you are willing to pay for my stock-market tips, I would sure hasten designing and perfecting my algorithms for the stock-market “prediction”s. … It just so happens that I had predicted yesterday, (Sunday 10th May) that the Bombay Stock Exchange’s Sensex indicator would definitely not rise today (on Monday), and that while the market should trade at around the same range as it did on Friday, the Sensex was likely to close a bit lower. This has come true. (It in fact closed just a bit higher than what I had predicted.)

… Of course, “one swallow does not a summer make.”… [Just checked it. Turns out that this one has come from Aristotle [^]. I don’t know why, but I had always carried the impression that the source was Shakespeare, or may be some renaissance English author. Apparently, not so.]

Still, don’t forget: If you have the money, I do have the inclination. And, the time. And, my data science skills. And, my algorithms. And…

…Anyway, take care and bye for now.


A song I like:

(Hindi) तुम से ओ हसीना कभी मुहब्बत मुझे ना करनी थी… (“tum se o haseenaa kabhee muhabbat naa” )
Music: Laxmikant Pyarelal
Singers: Mohammad Rafi, Suman Kalyanpur
Lyrics: Anand Bakshi

[I know, I know… This one almost never makes it to the anyone’s lists. If I were not to hear it (and also love it) in my childhood, it wouldn’t make to my lists either. But given this chronological prior, the logical prior too has changed forever for me. …

This song is big on rhythm (though they overdo it a bit), and all kids always like songs that emphasize rhythms. … I have seen third-class songs from aspiring/actual pot-boilers, songs like तु चीझ बडी है मस्त मस्त (“too cheez baDi hai mast, mast”), being a huge hit with kids. Not just a big hit but a huge one. And that third-rate song was a huge hit even with one of my own nephews, when he was 3–4 years old. …Yes,  eventually, he did grow up…

But then, there are some song that you somehow never grow out of. For me, this song is one of them. (It might be a good idea to start running “second-class,” why, even “third-class” songs, about which I am a bit nostalgic. I listened to a lot of them during this boring lock-down, and even more boring, during all those long-running trials for the MNIST dataset. The boredom, especially on the second count, had to be killed. I did. … So, all in all, from my side, I am ready!)

Anyway, once I grew up, there were a couple of surprises regarding the credits of this song. I used to think, by default as it were, that it was Lata. No, it turned out to be Suman Kalyanpur. (Another thing. That there was no mandatory “kar” after the “pur” also was a surprise for me, but then, I digress here.)

Also, for no particular reason, I didn’t know about the music director. Listening to it now, after quite a while, I tried to take a guess. After weighing in between Shankar-Jaikishan and R.D. Burman, also with a faint consideration of Kalyanji-Anandji, I started suspecting RD. …Think about it. This song could easily go well with those from the likes of तीसरी मंझील (“Teesri Manjhil”) or काँरवा (“karvaan”) right?

But as a self-declared RD expert, I also couldn’t shuffle my memory and recall some incidence in which I could be found boasting to someone in the COEP/IITM hostels that this one indeed was an RD song. … So, it shouldn’t be RD either. … Could it be Shankar-Jaikishan then?  The song seemed to fit in with the late 60s SJ mold too. (Recall Shammi.) … Finally, I gave up, and checked out the music director.

Well, Laxmikant Pyarelal sure was a surprise to me! And it should be, to any one. Remember, this song is from 1967. This was the time that LP were coming out with songs like those in दोस्ती (“Dosti”),  मिलन (“Milan”), उपकार (“Upakar”), and similar. Compositions grounded in the Indian musical sense, through and through.

Well yes, LP have given a lot songs/tunes that go more in the Western-like fold. (Recall रोज शाम आती थी (“roz shyaam aatee thee”), for instance.) Still, this song is a bit too “out of the box” for LP when you consider their typical “box”. The orchestration, in particular, also at times feels as if the SD-RD “gang” like Manohari Singh, Sapan Chakravorty, et al. wasn’t behind it, and so does the rendering by Rafi (espcially with that sharp हा! (“hah!”) coming at the end of the refrain)… Oh well.

Anyway, give a listen and see how you find it.]


Status: 99.78 % accuracy on the MNIST dataset (World Rank # 5, using a single CPU-only laptop)

As you know from my last post, for the past 5–6 weeks, I have been undertaking (very) extensive data-science trials. The one specific problem on which I have been working, during this time, is: the MNIST dataset.

About the MNIST problem:

Essentially, the MNIST problem is to teach a computer how to recognize hand-written digits. There are tons of materials on this topic, google on “MNIST” to know more.

MNIST has been called the “hello world” program of machine learning. The idea is that MNIST is merely a toy problem with which you begin exploring the vast area of Artificial Intelligence.

MNIST also has been described as the “drosophilia” of machine learning. IMO, this second characterization is more accurate. After all, OCR of digits is one problem that has kept an entire generation of PhD students and researchers burning oil at night!

Progress over time in the prediction accuracy on the MNIST dataset:

The accuracy in recognizing the MNIST digits has gone up from the earlier levels like some 80 % 20 years ago to 90+ % about 10 years ago, and then on to 99 % being routine today.

As the accuracy kept on increasing, to make reports easier on the eye (i.e., to avoid too many 9’s), people started reporting errors rather than correct results. Thus, instead of saying “I got 99.52 % accurate result”, people these days routinely just say: “I got 0.48% error rate.”

For testing your algorithms/models, the standardized test data provided by the MNIST dataset consists of exactly 10,000 samples. This well-rounded figure turns out to be quite convenient, because when you achieve, say, a 99.65 % accuracy level, you not only are 0.35 % wrong, but the actual number of samples on which you went wrong also is precisely 35. So, it’s very easy to convert between absolute error rates and percentage ones.

To come back to the progress in accuracy, with today’s libraries like PyTorch, TensorFlow and Keras, it indeed seems that MNIST has become a toy problem. Even a simple ‘net search will get you to hundreds of tutorials which show you “how to get into top 1 %”. [Sometimes, this is just a piece of hype which means nothing more than: how to get 99.0 % accuracynot how to get a 99-th percentile rank!

In other words, the trouble lies in the upper reaches of the accuracy scores. Each incremental progress from 99.65 % to 99.70 % to 99.75 % becomes harder and still harder—or even plain impossible to achieve, unless you basically improve your algorithm.

Despite throwing super-computers at it, no one has been able to achieve a 100.0% score on this dataset.

One reason for failing to achieve the 100% accuracy level is that some of the samples have come from people with astoundingly bad hand-writing.

Another reason for the failure is that a few of the standardized test samples (on which every one measures his accuracy) have in fact been wrongly labelled in the first place! For instance, the actual written-down digit may be “4”, but its standardized label says that it is “5” (because the guy who wrote that digit might have himself wrongly labelled it—or was a plain retard!). Now, once it’s standardized, if your algorithm actually guesses this sample right, then you lose on the accuracy as measured by the standardized data! This fact goes into the reason why it’s impossible to achieve 100.0% accuracy on the MNIST dataset.

Another trouble: There perhaps are too many ambiguous samples in this dataset. Two different algorithms may report the same accuracy (or error) level, say 26 errors. However, the particular samples on which they fail may not be exactly the same. In other words, there are many samples which show a tendency to get mis-classified on one algorithm but not on others!

Anyway, enough about these general remarks. Now, on to some hard data on the world-records.

The world’s best of the best:

While there are several Web pages that track the progress in the accuracy on the MNIST dataset, I found that many of the reliable ones have become outdated by now. In fact, today’s rank # 1 result does not appear in any of these compiled lists; I got to know of it through several Google searches.

So, here are the top 10 performances on the MNIST dataset, as found by me, as of today:

  • Rank # 1: Number of Errors: 16. Accuracy: 99.84%
    • Byerly, Kalganova and Dear. (31 Jan. 2020) “A branching and merging convolutional network with homogeneous filter capsules,” arXiv:2001.09136v3 [^]. Noted at “Papers with Code” [^]
  • Rank # 2: Number of Errors: 17. Accuracy: 99.83%
    • Assiri. (24 January 2020) “Stochastic optimization of plain convolutional neural networks with simple methods,” arXiv:2001.08856v1. [^] Noted at “Papers with Code” [^]
    • “Matuzas77”. (28 January 2020) Jupyter notebook at GitHub: [^]. Result noted at the Wiki [^].”
  • Rank # 3: Number of Errors: 18. Accuracy: 99.82%
    • Kowsari, Heidarysafa, Brown, Meimandi, Barnes. (2018) “RMDL: Random Multimodel Deep Learning for classification”, arXiv:1805.01890. [^] Noted at “Papers with Code” [^]
  • Rank # 4: Number of Errors: 21. Accuracy: 99.79%
    • Wan, Zeiler, Zhang, LeCun, and Fergus. (2013) [ PDF from LeCun’s site: ^]
  • Rank # 5: Number of Errors: 23. Accuracy: 99.77%
    • Cireşan, Meier, and Schmidhuber. (2012)
    • Sato, Nishimura, and Yokoi. (2015)
  • Rank # 6: Number of Errors: 24. Accuarcy: 99.76%
    • Chang, and Chen. (2015)
    • Bochinski, Senst, and Sikora. (2017)
    • Deotte.(2020) Rank # 1 on the Kaggle Leaderboard on the MNIST dataset.
      Noted on the Kaggle Leaderboard [^]. Comment: I suppose Deotte’s model carries over to the original MNIST dataset too, with at least this much accuracy. The two datasets differ. Kaggle has only 48 K of training data.
  • Rank # 7: Number of Errors: 25. Accuracy: 99.75%
    • Baldominos, Saez, and Isasi. (2018)
  • Rank # 8: Number of Errors: 27. Accuracy: 99.73%
    • Wan, Zeiler, Zhang, LeCun, and Fergus. (2013)
    • Cireşan, Meier, Gambardella, and Schmidhuber. (2011)
  • Rank # 9: Number of Errors: 28. Accuracy: 99.73%
    • Baldominos, Saez, and Isasi. (2019)
  • Rank # 10: Number of Errors: 28. Accuracy: 99.73%
    • Lee, Gallagher, and Tu. (2016)
    • Alom, Hasan, Yakopcic, and Taha. (2017)

Entries without explicit references to papers were obtained from the following source:

Alejandro Baldominos, Yago Saez, and Pedro Isasi, “A survey of handwritten character recognition with MNIST and EMNIST,” Appl. Sci. 2019, vol. 9, pp. 3169; doi:10.3390/app9153169 [^]

This source is more comprehensive and up to date (except for not including the entries for the top 3 ranks which came after the publication of the paper). Other sources, well-known but not always very up-to-date, are those by Prof. LeCun himself [^], Dr. Rodrigo Benenson [^], Benchmarks.AI [^], and the Wiki [^].

My current results (as of 13th April 2020, 11:00 IST):

Out of tens of trials conducted, let me pick out the two latest (and best) results which I’ve got so far:

  • Trials group 1 (my private code: 03.05)
    • Total training time: 23.3 hours (on my CPU-only machine)
    • Number of Errors: 24. Accuracy: 99.76 %.
    • Estimated world-rank: Together with the existing # 6.
  • Trials group 2 (my private code: 03.06)
    • Total training time: 27.5 hours (on my CPU-only machine)
    • Number of Errors: 22. Accuracy: 99.78 %.
    • Estimated world-rank: Above the existing rank # 5.

It must be noted that a significant reason for getting good accuracy levels is that I implement some new ideas of my own (parallels to which I have not seen anywhere else so far).

How do my results compare with the best in the world? Would it be possible to improve it?

Recall that I have a CPU-only machine. (I can’t buy a better machine because I don’t have any money, and in fact am in loans. I have been out of job for 2 years by now.)

I anticipate that if I were to have a GPU-enabled laptop, I should have crossed the 99.80 % accuracy level. This is not a vacuous statement; it comes with two different observation points:

Point No. 1:

Dr. Chris Deotte uses an ensemble of 15 models, each of which is run to 45 epochs. He gets to 99.75% accuracy. He is currently at # 1 position on the Kaggle leaderboard for MNIST.

Running Deotte’s one model with 45 epochs takes about 132 minutes on my machine. So, running an ensemble of 15 models would take about 1984 minutes, i.e., about 33 hours.

On the plus side, the Kaggle dataset has only 48 K training samples, and Deotte’s model still gets to 99.757 % accuracy (after averaging). It’s conceivable that with full 60 k training samples of the original MNIST, this architecture might go a few points higher.

On the minus side, when I ran Deotte’s architecture on my machine (as mentioned just above), I used the full 60 k of the original MNIST dataset. I still got only 99.70 % as the best accuracy. So, going much beyond 99.76 may not be possible, due to statistical nature of the training process.

To summarize this point:

33 hours on Deotte’s architecture might take you to 99.76 to 99.78 % accuracy, if you use the full 60 K training samples of the original MNIST dataset.

In comparison, my architecture (which makes use of Deotte’s insights) takes 24 to 28 hours, and definitely has gone up to 99.78 % accuracy already.

Point No. 2:

I have merely browsed through the architectures of the top 5 papers, not studied them. I presume that all of them use powerful parallel processing, at least GPUs, and that none uses a CPU-only machine (the way I do).

The top papers don’t usually give actual execution times required for training. However, one thing is striking.

The Rank # 1 paper uses a single model of about 1.5 million parameters, with which it goes up to 99.82 % (statistically best result). Using an ensemble of such models, it goes to 99.84 %.

In contrast, in my biggest architecture, each model of the ensemble was limited to fewer than half a million parameters (around 400 k).

The total number of parameters per model, taken by itself, is only an indicative measure, not conclusive. Details of the architecture matter, and so does the fine-tuning of a plethora of hyper-parameters. Processing differs. For instance, batch-normalization hardly adds any trainable parameters, but still results in a lot of extra training time (as well as some benefit for not overfitting or better accuracy).

To reach a conclusion on this point:

There is a definite margin to think that my new ideas might significantly enhance accuracy with even bigger architectures up to the tune of 1.5 million parameters.

Reasons for my present level of success, and future directions:

Let me highlight it again. I got (what to my mind are such) great results because of some new ideas that I thought of, and tried.

I mean to say, I tried everything “in the book,” and then some more. Thus, my model uses a deep network of CNNs, and ideas like batch normalization, dropout, data augmentation, and ensembling. That is to say, all the tricks that Dr. Chris Deotte explains so helpfully here [^] and puts to use here [^]. I used almost all of those tricks (or as many as I could, given my computing budget). And then, I added a few more tricks, based on my own ideas and thinking.

I have been abstractly thinking about some new ideas for image recognition for quite some time—for months, may be, and in one case, for almost 1.5 years or so. A group of closely related ideas from this brain-storming began acquiring a better shape late last year—though it was nothing immediately translatable into code. Then, I got some got to some fairly well concretized form in the first half of February 2020 (on the evening of the12th of that month, while in a restaurant, if what I remember is right).

However, in late February 2020, we had to make a move (i.e. shift everything to new house). A lot of time got “wasted” in it. So, I could start writing code only from 03 March 2020.

By 15th March 2020 (the date of my last post), it was already becoming increasingly clear to me that my ideas aren’t going to work out in their initial form. Yet, as I continued writing code and trying out improvements, these earlier ideas got transmutated into further new ideas. It is these latter ideas which were then implemented on bigger and various models, using TensorFlow and Keras.

There remain a few of these last group of ideas (the “transmutated” ones) which I do have in a fairly direct form, but which I could not actually try out so far, mainly because of the low computational power that I have. Trials take too much time, occupy the only machine that I have.

Trials are boring, in a way—while they are in progress. They are extremely boring, in fact, while they last. For hours and hours, you can do nothing but almost helplessly watch the stream of numbers appearing on the screen. You can’t even make out whether the idea you are trying is progressing for the better or the worse, because a handful of numbers at a time don’t give you any idea about the progress over several hours. Even if the current effort is actually worthless, you get to know only after hours, may be on the evening the next day. So, it’s all boring in the meanwhile, and tends to be draining in a way. That’s why, with 6 weeks spent like this, I am also nearing the fag-end of my patience.

Therefore, for now, I have decided to undertake just one more trial—a biggest one that I can still manage on my machine. I estimate that this single trial will go on for some 48  hours of run-time on my machine or more. Thanks to the TF-Keras, I can interrupt the trials once in a while, and resume training. So, it might take something like 4–5 days for this set of trial runs to get over.

I will see how this planned trial goes, and then will close this research on MNIST one way or the other. The reason is, I need to pursue other datasets like Fashion-MNIST, EMNIST, Not-MNIST, CIFAR-10, etc. too. My ideas need to be tested on those grounds too.

In the meanwhile, I also plan to update my resume soon enough, to include the current success on this research. (Yes, it more than amply qualifies for a full time engagement running for about 1.5 months!)

So, as far as MNIST is concerned, expect at the most one more blog post, if at all. In any case, for obvious reasons, I am going off the blog for another week.

One final clarification: I will not share the ideas or the code:

Take my report as being truthful or otherwise. I don’t care. I am not going to share the code. The ideas are patentable, and some of them are of more general nature. So, I am going to proceed very cautiously in sharing any details to any one. Sorry, but that’s how it is.

Credits and Acknowledgement:

While a whole range of materials by a lot of people helped me get here, let me single out three references/people over all others:

  • Dr. Michael Nielsen, for his online book on ANNs [^]. He expediated my learning time by a few factors over what would have happened if I were to use those usual university text-books. I got most of my ANN theory from him. Some of the ideas I now tried here had struck me, at least in the seed form, while learning ANNs from this book in 2018. (It’s good that I never shared those ideas back then!)
  • Dr. Jason Brownlee (again an Aussie!) [^], for his excellent online tutorials, and very simple and elegant toy code that helps get a feeling of having mastered some particular topic.
  • Dr. Chris Deotte, for his most excellent code [^], and even more helpful to me, his article detailing how he got to his Kaggle rank # 1 result [^]. I might have come to know about the best existing practices on CNNs and MNIST by consulting the original papers. But Deotte cut the time for people like me by months, perhaps even a couple of years, via just those two articles at Kaggle.

Thanks to all the three of them! And, thanks also to hundreds of people on different fora (like StackExchange) sharing insights, comments, fastest running code snippets, etc.

To conclude this post:

A poor jobless Indian man has achieved a 99.78% accuracy score on the MNIST dataset using a CPUs-only machine. He thus has an estimated rank of # 5 in the world today. His result has an accuracy better than that of the top-ranking kernel for the MNIST dataset on the Google-sponsored Kaggle’s leaderboard. He has further ideas to enhance accuracy, which, to his mind, are promising. However, he is limited by the available computation power of his machine. But in any case, he is not going to share any of the ideas or code with any one else.

Addendum: Also, the same man hopes that at least now, with this world-class achievement, the Indian/Foreign IT companies/recruiters/HR/Top Management idiots would stop asking him to take their tests before proceeding with any employment consideration. If they can manage to get their “high-quality” people / geniuses with their tests and without reference to world-class achievements, well and good for them; he wishes them further “success.”

A song I like:

(Hindi) पीछे पीछे आ कर, छू लो हमें पा कर (“peechhe peechhe aa kar, chhoo lo hame paa kar”)
Singers: Lata Mangeshkar, Hemant Kumar
Music: S. D. Burman
Lyrics: Sahir Ludhianvi

[This song has a very fresh feel overall—just like so many songs that came from the Dev Anand + SD Burman combination. I don’t know why this song escaped me for such a long time while writing the songs section. (I’ve been listening to it off and on. It’s just that while actually writing the songs section, this song didn’t strike me, that’s all)

… As an exception, I can recommend the video too. I liked the picturization here, though the photography is techically not so sound (the contrast is not well managed). But it’s Mahabaleshwar—my favorite area. Kalpana Kartik’s dancing (jumping around, really speaking!) is so natural. Hardly anything of today’s will ever even approach this kind of an utter naturalness, this kind of a freshness. BTW, to those who don’t know, Kalpana Kartik and Dev Anand were husband and wife in real life. I guess they had already married (1954) by the time this movie was shot (released 1955). … So, this song is like 65 years old!…

Anyway, stay at home, wear mask (the droplet sizes are relatively big so masks can definitely help, and even the air-flow dynamics around a single virus particle, at that small a scale, should be almost Brownian, not of Navier-Stokes), wash your hands, and engage your mind in something else… Should be back after after a week or 10 days. Take care and bye for now…]