How do I auto-scroll and play a custom audio

[annetic]

It has been confusing for me to experiment with verovio. How can I auto-scroll and also play a custom audio instead of midi.

[ahankinson]

What have you tried?

[annetic]

Tried out many options, looked at code of other websites, became confusing.

[craigsapp]

Do you have your own interface to verovio or are you using an existing one? And in what language (presumably Javascript)? Basically you need to ask the HTML5 element <audio> element what is the current time of playback is, and then use the timemap from the verovio toolkit to manage note highlighting and scrolling (there are various ways of handling that).

If you render a MIDI file into a sound file, then the millisecond timings in the timemap (tstamp values) are all you need. This is done on the Tasso in Music Project website, for example:
https://www.tassomusic.org/work/?id=Trm0047m
The playback button is playing an MP3 soundfile that was created offline from a MIDI file (because I wanted slightly better sound than just a piano timbre for all voices). You can also click on a note in the score to start playing the MP3 file from that note.

If you want to use a sound file that is a live performance, then you will have to create an additional performance timeline that @ahankinson could explain how to embed in the source MEI file, or you would have a separate event list to manage the mapping of performance time to score time that the verovio timemap provides.

Related to this: @lpugin, is there an option to output the output the calculated timemap from verovio on the command line? I though there was one, but I do not see it in the list of verovio options. I can add an issue to add it (back?) if you like 😉

[annetic]

The timemap option seems nice for custom audio playback. But also MXL files, could I use it with this?

[lpugin]

Yes

[craigsapp]

is there an option to output the output the calculated timemap from verovio on the command line?

Yes, it was snuck into the -t option, so on the command line it is:

verovio -t timemap file.mei

and the output is saved in file.json

for the javascript toolkit it would be the option setting {"outputTo": "timemap"}

[craigsapp]

I have a simpler implementation for alignment with a live performance that I will look for (I implemented it in 2016, so we will see if it still works when I find it).

[annetic]

Okay, how to timemap the audio file after I do {"outputTo": "timemap"}

[craigsapp]

I use Sonic Vlsualiser to extract timings of note onsets in a semi automated way (which I can describe in more detail if needed). Related to Sonic Visualiser is a newer program called Sonic Lineup, which I have not used yet, but seems potentially interesting for you. Probably you would render a MIDI file with known timings into an Audio file, and then align that audio file with live performances to get the event timings in the performance audio files.

There are other automated ways (but I use Sonic Visualiser in a semiautomated way for more accuracy). Of note is:

Nakamura, E., Yoshii, K., Katayose, H. (2017). Performance Error Detection and Post-Processing for Fast and
Accurate Symbolic Music Alignment. In ISMIR (pp. 347-353).

Website with software: https://midialignment.github.io/demo.html

This one might be harder to works with audio, since it is more MIDI to score alignment. @wergo works with this for MIDI, but he might know of any automatic audio-to-midi alignment software?

[annetic]

Thank you, Is there any sample code I could experiment on?