# Sound separation, voice separation from a song, the cocktail party effect, etc., and AI/ML

A Special note for the Potential Employers from the Data Science field:

Recently, in April 2020, I achieved a World Rank # 5 on the MNIST problem. The initial announcement can be found here [^], and a further status update, here [^].

All my data science-related posts can always be found here [^].

1. The “Revival” series of “Sa Re Ga Ma” music company:

It all began with the two versions of the song which I ran the last time (in the usual songs section). The original song (or so I suppose) is here [^]. The “Revival” edition of the same song is here [^]. For this particular song, I happened to like the “Revival” version just so slightly better. It seemed to “fill” the tonal landscape of this song better—without there being too much of degradation to the vocals, which almost always happens in the “Revival” series. I listened to both these versions back-to-back, a couple of times. Which set me thinking.

I thought that, perhaps, by further developing the existing AI techniques and using them together with some kind of advanced features for manual editing, it should be possible to achieve the same goals that the makers of “Revival” were aiming at, but too often fell too short of.

So, I did a bit of Google search, and found discussions like this one at HiFiVision [^], and this one at rec.music [^]. Go through the comments; they are lively!

On the AI side, I found this Q&A at Quora: [^]. One of the comments said: “Just Google this for searching `software to clarify and improve sound recordings’ and you will have several thousand listings for software that does this job.”

Aha! So there already were at least a few software to do the job?

A few more searches later, I landed at Spleeter by Deezer [^]. (In retrospect, this seems inevitable.)

2. AI software for processing songs: Spleeter:

Spleeter uses Python and TensorFlow, and the installation instructions assume Conda. So, I immediately tried to download it, but saw my Conda getting into trouble “solving” for the “environment”. Tch. Too bad! I stopped the Conda process, closed the terminal window, and recalled the sage advice: “If everything fails, refer to the manual.” … Back to the GitHub page on Deezer, and it’s only now that I find that they still use TF 1.x! Bad luck!

Theoretically, I could have run Spleeter on Google Colab. But I don’t like Google Colab. Nothing basically wrong with it, except that (i) personally, I think that Jupyter is a great demo tool but a very poor development tool, and (ii) I like my personal development projects unclouded. (I will happily work on a cloud-based project if someone pays me to do it. For that matter, if someone is going to pay me and then also advices me to set up the swap space for my local Ubuntu installation on the cloud, I will happily do it too! But when it comes to my own personal projects, don’t expect me to work on the cloud.)

So, tinkering around with Spleeter was, unfortunately, not possible. But I noticed what they add at the Spleeter’s page: “there are multiple forks exposing spleeter through either a Guided User Interface (GUI) or a standalone free or paying website. Please note that we do not host, maintain or directly support any of these initiatives.” Good enough for me. I mean, the first part…

So, after a couple of searches, I landed at a few sites, and actually tried two of them: Melody.ml [^] and Lala.ai [^]. The song I tried was the original version of the same song I ran the last time (links to YouTube given above). I went for just two stems in both cases—vocals and the rest.

Ummm…. Not quite up to what I had naively expected. Neither software did too well, IMO. Not with this song.

3. Peculiarity of the Indian music:

The thing is this: Indian music—rather, Indian film (and similar) music—tends to be very Indian in its tastes.

We Indians love curries—everything imaginable thrown together. No 5 or 7 or 9 or 11 course dinners for us. All our courses are packed into a big simultaneous serving called “thaali”. Curries are a very convenient device in accomplishing this goal of serving the whole kitchen together. (BTW, talking of Western dinners, it seems to me that they always have an odd number of courses in dinners. Why so? why not an even number of courses? Is it something like that 13 thing?)

Another thing. Traditionally, Indian music has made use of many innovations, but ideas of orchestra and harmony have never been there. Also, Indian music tends to emphasize continuity of sounds, whereas Western music seems to come in a “choppy” sort of style. In Western music, there are clear-cut demarcations at every level, starting from very neatly splitting a symphony into movements, right down to individual phrases and the smallest pieces of variations. Contrast it all with the utter fluidity and flexibility with which Indian music (especially the vocal classical type) gets rendered.

On both counts, Indian music comes across as a very fluid and continuous kind of a thing: a curry, really speaking.

All individual spices (“masaalaa” items) are not just thrown together, they are pounded and ground together into a homogenous ball first. That’s why, given a good home-made curry, it is tough to even take a good guess at exactly what all spices might have gone into making it. …Yes, even sub-regional variations are hard to figure out, even for expert noses. Just ask a lady from North Maharashtra or Marathwada to name all the spices used in a home-made curry from the north Konkan area of Maharashtra state (which is different from the Malavani sub-region of Konkan), going just by the taste. These days, communications have improved radically, and so, people know the ingredients already. But when I was young, women of my mother’s age would typically fail to guess the ingredients right. So, curries come with everything utterly mixed-up together. The whole here is not just greater than the sum of its parts; the whole here is one whose parts cannot even be teased out all that easily.

In India, our songs too are similar to curries. Even when we use Western instruments and orchestration ideas, the usage patterns still tend to become curry-like. Which means, Indian songs are not at all suitable for automatic/mechanically conducted analysis!

That’s why, the results given by the two services were not very surprising. So, I let it go at that.

But the topic wasn’t prepared to let me go so easily. It kept tugging at my mind.

4. Further searches:

Today, I gave in, and did some further searches. I found Spleeter’s demo video [^]. Of course, there could be other, better demo’s too. But that’s what I found and pursued. I also found a test of Spleeter done on Beatles’ song: “Help” [^].

Finally, I also found this video which explains how to remove vocals from a song using Audacity[^]. Skip it if you wish, but it was this video which mentioned Melodyne [^], which was a new thing to me. Audacity is Open Source, whereas Melodyne is a commercial product.

Further searches later, I also found this video (skip it if you don’t find all these details interesting), using ISSE [^]. Finally, I found this one (and don’t skip it—there’s a wealth of information in it): [^]. Among many things, it also mentions AutoTune [^], a commercial product. Google search suggested AutoTalent as its Open Source alternative; it was written by an MIT prof [^]. I didn’t pursue it a lot, because my focus was on vocals-extraction rather than vocals pitchcorrection.

Soooooo, where does that leave us?

Without getting into all the details, let me just state a few conclusions that I’ve reached…

5. My conclusions as of today:

5.1. Spleeter and similar AI/ML-based techniques need to improve a lot. Directly offering voice-separation services is not likely to take the world by the storm.

5.2. Actually, my position is better stated by saying this: I think that directly deploying AI/ML in the way it is being deployed, isn’t going to work out—at all. Just throwing tera-bytes of data at the problem isn’t going to solve it. Not because the current ML techniques aren’t very capable, but because music is far too complex. People are aiming for too high-hanging a fruit here.

5.3. At the same time, I also think that in a little more distant future, say over a 5–10 years’ horizon, chances are pretty good that tasks like separating the voice from the instrumental sounds would become acceptably good. Provided, they pursue the right clues.

6. How the development in this field should progress (IMO):

In this context, IMO, a good clue is this: First of all, don’t start with AI/ML, and don’t pursue automation. Instead, start with a very good idea of what problem(s) are at all to be solved.

In the present context, it means: Try to understand why products like Melodyne and AutoTune have at all been successful—despite there being “so little automation” in them.

My answer: Precisely because these software have given so much latitude to the user.

It’s only after understanding the problem to be solved, and the modalities of current software, that we should come to this question of whether the existing capabilities of these software can at all be enhanced using AI/ML, using one feature/aspect at a time.

My answer, in short: Yes, they can (and should) be.

Notice, we don’t start with the AI/ML algorithms and then try to find applications for them. We start with some pieces of good software that have already created certain needs (or expanded them), and are fulfilling them already. Only then do we think of enhancing it—with AI/ML being just a tool. Yes, it’s an enabling technology. But in the end, it’s just a tool to improve other software.

Then, as the next step, consolidate all possible AI-related gains first—doing just enhancements, really speaking. Only then proceed further. In particular, don’t try to automate everything right from the beginning.

IMO, AI/ML techniques simply aren’t so well developed that they can effectively tackle problems involving relatively greater conceptual scope, such that wide-ranging generalizations get involved in complex ways, in performing a task. AI/ML techniques can, and indeed do, excel—and even outperform people—but only in those tasks that are very narrowly defined—tasks like identifying handwritten digits, or detecting few cancerous cells from among hundreds of healthy cells using details of morphology—without ever getting fatigued. Etc.

Sound isolation is not a task very well suited to these algorithms. Not at this stage of development of AI/ML, and the sound-related softwares.

“The human element will always be there,” people love to repeat in the context of AI.

Yes, I am an engineer and I am very much into AI/ML. But when it comes to tasks like sound separation and all, my point is stronger than what people say. IMO, it would be actually stupid to throw away the manual editing aspects.

Human ear is too sensitive an instrument; it takes almost nothing (and almost no time) for most of us to figure out when some sound processing goes bad, or when a reverse-FFT’ed sound begins to feel too shrill at times, or too “hollow” at other times, or plain “helium-throat”-like at still others [^][^].

Gather some 5–10 people in a room and play some songs on a stereo system that is equipped with a good graphic equalizer. If there is no designated DJ, what is the bet that people are just going to fiddle around with the graphic equalizer every time a new song begins? The reason is not just that people want to impress others with their expertize. The fact of the matter is, people are sensitive to even minutest variations that come in sound, and they will simply not accept something which does not sound “just right.” Further, there are individual tastes too—as to what precisely is “just right”. That’s why, if one guy increases the bass, someone else is bound to get closer to the graphic equalizer to set it right! It’s going to happen.

That’s why, it’s crucial not to even just attempt to “minimize” the human “interference.” Don’t do that. Not for software like these.

Instead, the aim should be to keep that human element right at the forefront, despite using AI/ML.

Ditto, for other similarly complex tasks / domains, like colouring B&W images, generating meaningful passages of text, etc.

That’s what I think. As of today.

7. Guess I also have some ideas for processing of music:

So, yes, I am not at all for directly starting training Deep Learning models with lots of music tracks.

At the same time, I guess, I also have already started generating quite a few ideas regarding these topics: analysis of music, sound separation, which ML technique might work out well and in what respect (for these tasks), what kind of abstract information to make available to the human “operator” and in what form/presentation, etc. …

…You see, some 15+ years ago, I had actually built a small product called “ToneBrush.” It offered real-time visualizations of music using windowed FFT and further processing (something like spectrogram and the visualizations which by now have become standard in all media players like VLC etc.). My product didn’t even sell just a single copy! But it was a valuable experience…

…Cutting back to the present, all the thinking which I did back then, now came back to once again. … All the same, for the time being, I’m just noting these ideas in my notebook, and otherwise moving this whole topic on to the back-burner. I first want to finish my ongoing work on QM, first of all.

One final note, an after-thought, actually: I didn’t say anything about the cocktail party effect. Well, if you don’t know what the effect means, start with the Wiki [^]. As to its relevance: I remember seeing some work (I think it was mentioned at Google’s blog) which tried to separate out each speaker’s voice from the mixed up signals coming from, say, round-table discussion kind of scenarios. However, I am unable to locate it right now. So, let me leave it as “homework” for you!

8. On the QM front:

As to my progress on the QM side—or lack of it: I spotted (and even recalled from memory) quite a few more conceptual issues, and have been working through them. The schedule might get affected a bit, but not a lot. Instead of 3–4 weeks which I announced 1–2 weeks ago, these additional items add a further couple of weeks or so, but not more. So, instead of August-end, I might be ready by mid-September. Overall, I am quite happy with the way things are progressing in this respect. However, I’ve realized that this work is not like programming. If I work for more than just 7–8 hours a day, then I still get exhausted. When it’s programming and not just notes/theory-building, then I can easily go past 12 hours a day, consistently, for a lot longer period (like weeks at a time). So, things are moving more slowly, but quite definitely, and I am happy with the progress so far. Let’s see.

In the meanwhile, of course, thoughts on topics like coloring of B&W pics or sound separation also pass by, and I note them.

OK then, enough is enough. See you after 10–15 days. In the meanwhile, take care, and bye for now…

A song I like:

(Western, Pop) “Rhiannon”
Band: Fleetwood Mac

[I mean this version: [^]. This song has a pretty good melody, and Stevie Nicks’s voice, even if it’s not too smooth and mellifluous, has a certain charm to it, a certain “femininity” as it were. But what I like the most about this song is, once again, its sound-scape taken as a whole. Like many songs of its era, this song too carries a right level of richness in its tonal land-scape—neither too rich nor too sparse/rarefied, but just right. …If I recall it right, surprisingly, the first time I heard this song was not in the COEP hostels, but in the IIT Madras hostels.]

History:
— 2020.08.09 02:43 IST: First published
— 2020.08.09 20:33 IST: Fixed the wrong linkings and the broken links, and added reference to AutoTalent.

# The rule of omitting the self-field in calculations—and whether potentials have an objective existence or not

There was an issue concerning the strictly classical, non-relativistic electricity which I was (once again) confronted with, during my continuing preoccupation with quantum mechanics.

Actually, a small part of this issue had occurred to me earlier too, and I had worked through it back then.

However, the overall issue had never occurred to me with as much of scope, generality and force as it did last evening. And I could not immediately resolve it. So, for a while, especially last night, I unexpectedly found myself to have become very confused, even discouraged.

Then, this morning, after a good night’s rest, everything became clear right while sipping my morning cup of tea. Things came together literally within a span of just a few minutes. I want to share the issue and its resolution with you.

The question in question (!) is the following.

Consider 2 (or $N$) number of point-charges, say electrons. Each electron sets up an electrostatic (Coulombic) potential everywhere in space, for the other electrons to “feel”.

As you know, the potential set up by the $i$-th electron is:
$V_i(\vec{r}_i, \vec{r}) = \dfrac{1}{4 \pi \epsilon_0} \dfrac{Q_i}{|\vec{r} - \vec{r}_i|}$
where $\vec{r}_i$ is the position vector of the $i$-th electron, $\vec{r}$ is any arbitrary point in space, and $Q_i$ is the charge of the $i$-th electron.

The potential energy associated with some other ($j$-th) electron being at the position $\vec{r}_j$ (i.e. the energy that the system acquires in bringing the two electrons from $\infty$ to their respective positions some finite distance apart), is then given as:
$U_{ij}(\vec{r}_i, \vec{r}_j) = \dfrac{1}{4 \pi \epsilon_0} \dfrac{Q_i\,Q_j}{|\vec{r}_j - \vec{r}_i|}$

The notation followed here is the following: In $U_{ij}$, the potential field is produced by the $i$-th electron, and the work is done by the $j$-th electron against the $i$-th electron.

Symmetrically, the potential energy for this configuration can also be expressed as:
$U_{ji}(\vec{r}_j, \vec{r}_i) = \dfrac{1}{4 \pi \epsilon_0} \dfrac{Q_j\,Q_i}{|\vec{r}_i - \vec{r}_j|}$

If a system has only two charges, then its total potential energy $U$ can be expressed either as $U_{ji}$ or as $U_{ij}$. Thus,
$U = U_{ji} = U_{ij}$

Similarly, for any pair of charges in an $N$-particle system, too. Therefore, the total energy of an $N$-particle system is given as:
$U = \sum\limits_{i}^{N} \sum\limits_{j = i+1}^{N} U_{ij}$

The issue now is this: Can we say that the total potential energy $U$ has an objective existence in the physical world? Or is it just a device of calculations that we have invented, just a concept from maths that has no meaningful physical counterpart?

(A side remark: Energy may perhaps exist as an attribute or property of something else, and not necessarily as a separate physical object by itself. However, existence as an attribute still is an objective existence.)

The reason to raise this doubt is the following.

When calculating the motion of the $i$-th charge, we consider only the potentials $V_j$ produced by the other charges, not the potential produced by the given charge $V_i$ itself.

Now, if the potential produced by the given charge ($V_i$) also exists at every point in space, then why does it not enter the calculations? How does its physical efficacy get evaporated away? And, symmetrically: The motion of the $j$-th charge occurs as if $V_j$ had physically evaporated away.

The issue generalizes in a straight-forward manner. If there are $N$ number of charges, then for calculating the motion of a given $i$-th charge, the potential fields of all other charges are considered operative. But not its own field.

How can motion become sensitive to only a part of the total potential energy existing at a point even if the other part also exists at the same point? That is the question.

This circumstance seems to indicate as if there is subjectivity built deep into the very fabric of classical mechanics. It is as if the universe just knows what a subject is going to calculate, and accordingly, it just makes the corresponding field mystically go away. The universe—the physical universe—acts as if it were changing in response to what we choose to do in our mind. Mind you, the universe seems to change in response to not just our observations (as in QM), but even as we merely proceed to do calculations. How does that come to happen?… May be the whole physical universe exists only in our imagination?

Got the point?

No, my confusion was not as pathetic as that in the previous paragraph. But I still found myself being confused about how to account for the fact that an electron’s own field does not enter the calculations.

But it was not all. A non-clarity on this issue also meant that there was another confusing issue which also raised its head. This secondary issue arises out of the fact that the Coulombic potential set up by any point-charge is singular in nature (or at least approximately so).

If the electron is a point-particle and if its own potential “is” $\infty$ at its position, then why does it at all get influenced by the finite potential of any other charge? That is the question.

Notice, the second issue is most acute when the potentials in question are singular in nature. But even if you arbitrarily remove the singularity by declaring (say by fiat) a finite size for the electron, thereby making its own field only finitely large (and not infinite), the above-mentioned issue still remains. So long as its own field is finite but much, much larger than the potential of any other charge, the effects due to the other charges should become comparatively less significant, perhaps even negligibly small. Why does this not happen? Why does the rule instead go exactly the other way around, and makes those much smaller effects due to other charges count, but not the self-field of the very electron in question?

While thinking about QM, there was a certain point where this entire gamut of issues became important—whether the potential has an objective existence or not, the rule of omitting the self-field while calculating motions of particles, the singular potential, etc.

The specific issue I was trying to think through was: two interacting particles (e.g. the two electrons in the helium atom). It was while thinking on this problem that this problem occurred to me. And then, it also led me to wonder: what if some intellectual goon in the guise of a physicist comes along, and says that my proposal isn’t valid because there is this element of subjectivity to it? This thought occurred to me with all its force only last night. (Or so I think.) And I could not recall seeing a ready-made answer in a text-book or so. Nor could I figure it out immediately, at night, after a whole day’s work. And as I failed to resolve the anticipated objection, I progressively got more and more confused last night, even discouraged.

However, this morning, it all got resolved in a jiffy.

Would you like to give it a try? Why is it that while calculating the motion of the $i$-th charge, you consider the potentials set up by all the rest of the charges, but not its own potential field? Why this rule? Get this part right, and all the philosophical humbug mentioned earlier just evaporates away too.

I would wait for a couple of days or so before coming back and providing you with the answer I found. May be I will write another post about it.

Update on 2019.03.16 20:14 IST: Corrected the statement concerning the total energy of a two-electron system. Also simplified the further discussion by couching it preferably in terms of potentials rather than energies (as in the first published version), because a Coulombic potential always remains anchored in the given charge—it doesn’t additionally depend on the other charges the way energy does. Modified the notation to reflect the emphasis on the potentials rather than energy.

A song I like:

[What else? [… see the songs section in the last post.]]
(Hindi) “woh dil kahaan se laaoon…”
Singer: Lata Mangeshkar
Music: Ravi
Lyrics: Rajinder Kishen

A bit of a conjecture as to why Ravi’s songs tend to be so hummable, of a certain simplicity, especially, almost always based on a very simple rhythm. My conjecture is that because Ravi grew up in an atmosphere of “bhajan”-singing.

Observe that it is in the very nature of music that it puts your mind into an abstract frame of mind. Observe any singer, especially the non-professional ones (or the ones who are not very highly experienced in controlling their body-language while singing, as happens to singers who participate in college events or talent shows).

When they sing, their eyes seem to roll in a very peculiar manner. It seems random but it isn’t. It’s as if the eyes involuntarily get set in the motions of searching for something definite to be found somewhere, as if the thing to be found would be in the concrete physical space outside, but within a split-second, the eyes again move as if the person has realized that nothing corresponding is to be found in the world out there. That’s why the eyes “roll away.” The same thing goes on repeating, as the singer passes over various words, points of pauses, nuances, or musical phrases.

The involuntary motions of the eyes of the singer provide a window into his experience of music. It’s as if his consciousness was again and again going on registering a sequence of two very fleeting experiences: (i) a search for something in the outside world corresponding to an inner experience felt in the present, and immediately later, (ii) a realization (and therefore the turning away of the eyes from an initially picked up tentative direction) that nothing in the outside world would match what was being searched for.

The experience of music necessarily makes you realize the abstractness of itself. It tends to make you realize that the root-referents of your musical experience lie not in a specific object or phenomenon in the physical world, but in the inner realm, that of your own emotions, judgments, self-reflections, etc.

This nature of music makes it ideally suited to let you turn your attention away from the outside world, and has the capacity or potential to induce a kind of a quiet self-reflection in you.

But the switch from the experience of frustrated searches into the outside world to a quiet self-reflection within oneself is not the only option available here. Music can also induce in you a transitioning from those unfulfilled searches to a frantic kind of an activity: screams, frantic shouting, random gyrations, and what not. In evidence, observe any piece of modern American / Western pop-music.

However, when done right, music can also induce a state of self-reflection, and by evoking certain kind of emotions, it can even lead to a sense of orderliness, peace, serenity. To make this part effective, such a music has to be simple enough, and orderly enough. That’s why devotional music in the refined cultural traditions is, as a rule, of a certain kind of simplicity.

The experience of music isn’t the highest possible spiritual experience. But if done right, it can make your transition from the ordinary experience to a deep, profound spiritual experience easy. And doing it right involves certain orderliness, simplicity in all respects: tune, tone, singing style, rhythm, instrumental sections, transitions between phrases, etc.

If you grow up listening to this kind of a music, your own music in your adult years tends to reflect the same qualities. The simplicity of rhythm. The alluringly simple tunes. The “hummability quotient.” (You don’t want to focus on intricate patterns of melody in devotional music; you want it to be so simple that minimal mental exertion is involved in rendering it, so that your mental energy can quietly transition towards your spiritual quest and experiences.) Etc.

I am not saying that the reason Ravi’s music is so great is because he listened his father sing “bhajan”s. If this were true, there would be tens of thousands of music composers having talents comparable to Ravi’s. But the fact is that Ravi was a genius—a self-taught genius, in fact. (He never received any formal training in music ever.) But what I am saying is that if you do have the musical ability, having this kind of a family environment would leave its mark. Definitely.

Of course, this all was just a conjecture. Check it out and see if it holds or not.

… May be I should convert this “note” in a separate post by itself. Would be easier to keep track of it. … Some other time. … I have to work on QM; after all, exactly only half the month remains now. … Bye for now. …