[how not to] convert Audio to Sheetmusic or MIDI
Tutorial now Research Paper
Home Too long didn't read (summary)This article was viewed: <12> times
A friend asked me to convert a folk song into sheet music... on the surface that looked easy so I agreed. Proceeds to fall down a rabbit hole on audio conversion and random neural networks... Its weird.
Now the task of converting audio data to sheet music namely XML, so it can be imported into programs like Sibelius and Musescore is... REALLY HARD :0, but after experimenting for a few days I found a few techniques you can use so you can get a moderately clean manuscript.
Tips For Recording Audio
- Record with minimal noise
- if possible, record multiple streams for your different instruments
- If you're converting music, convert music that have well pronounced instruments, so if you can find a piano cover as they can be easily detected. Instruments like the violin as they don't move in steps can confuse the computer as their frequencies have a bell curve slope.
- Chose genres that have a stable and beat and Rhythm so Pop and indie music works well but Rock and Folk are really hard to convert as so much is going on, and they have many key and time signature changes
Audio Seperation
If you're using already recorded music try separating the tracks by instrument either manually or using an Ai service I used lalal.ai/ I recommend separating the tracks for a cleaner conversion if you don't the computer will 100% get confused and start using notes from other instruments and make you manuscript confusing
Audio Examples
Example 1a: Without Isolation
music: Drunk at Night by something fiddles i dunno ¯\_(ツ)_/¯ (jk its Blazin' Fiddles)Example 1b: With Violin Isolation
Example 1c: With Piano Isolation
Example: 2a Piano Isolation
music: Box Of Ashes by: Ian Hubert and the Freeway Pirates [in the short film series dynamo dream] (its an amazing song from: 0:50 is amazing)Example: 2b Voice Isolation
What I think the software is doing is detecting the voices or intrument and just removing any of the frequencies that do not follow the same pattern, kind of like find the odd one out but instead of shapes or people.... its numbers.
I tried the same with a AUDACITY and it worked well use the remove the noise reduction effect and get a noise profile on anything you want to remove to isolate a track.
After this I looked for services that convert audio to sheet music I did find one in the end but at the time of when I did it I thought it would be smart to convert it to MIDI then to a Manuscript... You can do this, but I would not recommend it. Use a program called AnthemScore it has a 30-day free trial which is what I used (I ain't paying for software) which converts the audio to either MIDI or sheet music
But I'm stupid and didn't do this, which opened another can of worms... I chose to convert the audio to MIDI by using a product of Spotify namely Basic Pitch: An open source MIDI converter the online tool is very, VERY slow so if you're a nerd (like me lmao) you can run the source code on your computer which is what I did for slightly more control. Now the results that Basic Pitch gives are better than AnthemScore, but it takes forever, and I have little to no control on how the software should detect and interpret the audio and frequencies. If your trying to get the notes for a violin (or any instument for that matter) limit the base frequencies to "195 HZ" which is the lowest note on the violin (which is G) to remove low and random frequencies (change the frequency to the lowest of your desired instrument).
this is an image of the frequencies in a spectogram. Yellow is the exact frequencies and Red is the average frequencies. As you can see there is a curve meaning that when a note is played the HZ increases then decreases, this can be miss read as multiple notes and confuse basicpitch (just something to take note of)
Once I got the midi file I opened it in a software called CakeWalk which is a free DAW that I have used in the past, I used this to clean up the MIDI file (you can use any MIDI editor, there are also some online ones)
Now import it into your Manuscript software I used Musescore because it's free and OPEN-SOURCE, at the time of writing Musescore 4 does not have a midi importer (due to M4 being rewritten), it can import Midi files, but there is no Dialogue box that has import options which is needed in this case, so I'd recommend rolling back to Musescore 3 as it has the Midi importer (and Musescore 3 is just plain better than 4 [at this moment]) I recommend checking and unchecking these boxes
.
- Max Voices = 1
- [x] Halving The Bar Count
- [x] Is Human performance
- [x] Split stave (leave unchecked if your instrument has 2 staves)
- [ ] Clef Changes (leave checked if your instrument needs a bass clefs and treble or any of the other clefs that no ones cares about [looking at you alto JK])
- [x] Simplify Durations
- [x] Show Staccato
- [x] Dotted Notes
- [x] Show Tempo Text
Play around with quantizing your score as Musescore likes adding random rests, quantizing can help remove some of them quaver notes (eights) are OK for quantizing.
Final Results
Manuscript
Original
Concluding and closing words:
And that's it.... Just use AnthemScore save your time!!
Now you probably thinking... well the sheetmusic sounds a bit crap... and thats cause it is, I could have cleaned up the MIDI file before converting to Sheetmusic but well I didnt know how to and didnt have the time ( the whole piece is 6min :0 ) now you could have done this a lot better and a lot more efficiently but as a test of a really new way of working with audio files and sheetmusic in general, I would call this....
Still a complete failure 😅
Byeeee
EDITED: I forgot to mention melodyne which could give you better conversion but i didnt have the budget to pay for a new peice of software so results will vary.
whatever.green 13-04-23
.....
...
.