Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 13 of 13

Thread: Transform an acoustic harp into a midi harp

  1. #1
    Junior Member
    Join Date
    Mar 2020
    Posts
    4

    Transform an acoustic harp into a midi harp

    Good morning, sir,
    I'm new to the forum and pretty new to Teensy.
    I've only done small projects so far.
    I'd like some advice on the feasibility of a project.
    We have an acoustic harp that we want to detect the frequency of each of the 34 strings and send the corresponding Midi commands.
    I have already done tests with other products (arduino or other) but without success.
    While searching on the internet, I discovered the Teensy which seems well adapted to such a project.
    But before consuming certainly hours and hours on the subject, I would like to know if it seems feasible to you.
    To sum up:
    * I have to detect whether or not one of the 34 strings has been pinched.
    * then determine its frequency and the amplitude of the vibration.
    * send all this information via the midi protocol

    I have a teensy 3.6 and a teensy 4.0.

    Do you think that's feasible?
    Do you have any suggestions or ideas for me to get started?
    Is it possible to use the 'audio design tool'?

    Thank you very much for your answers.

  2. #2
    Senior Member ETMoody3's Avatar
    Join Date
    Mar 2014
    Location
    New Ulm, Mn
    Posts
    141
    It's feasible, but there's a little lag/delay that may not work well for you when using frequency detection and converting to midi data. I have indeed done this myself with aTeensy 4 and find the performance somewhat less than musically useful. The lag is about 20 milliseconds, sometimes more. Further more the frequency detection doesn't exactly work on clusters of notes per se, the algorithm delivers a frequency that is related but generally lower by an octave or two.

    Here's some audio of the frequency detection in the audio library working on a guitar input and triggering synthesis directly in a Teensy 4 :

    https://youtu.be/ROiPyU2PGRM

    It may be more possible to physically detect the pluck of any string and with the pitch of any given string being a known value, simply send the midi data corresponding.
    This would in theory work faster than pitch detection and have the added benefit of polyphony.

    As for hardware to detect each string.... not sure, many possibilities. Many. Piezo sensors, infrared rangefinders, accelerometers, many possibilities most of which are probably bad ideas...(for lots of reasons starting with mutilation of the poor harp to hang sensors off of it. )

    The "Audio design tool" simply assists you in creating routing code, it doesn't create any functions.

  3. #3
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,352
    yeah... from audio you are unlikely to get the results you are after. Polyphony would be somewhere between very tricky and nearly impossible.

    A MIDI harp with detectors on each string wouldn't be very difficult (to code) but might be to build especially if you want it to still work as an acoustic instrument.

  4. #4
    Junior Member
    Join Date
    Mar 2020
    Posts
    4
    Quote Originally Posted by ETMoody3 View Post
    It's feasible, but there's a little lag/delay that may not work well for you when using frequency detection and converting to midi data. I have indeed done this myself with aTeensy 4 and find the performance somewhat less than musically useful. The lag is about 20 milliseconds, sometimes more. Further more the frequency detection doesn't exactly work on clusters of notes per se, the algorithm delivers a frequency that is related but generally lower by an octave or two.

    Here's some audio of the frequency detection in the audio library working on a guitar input and triggering synthesis directly in a Teensy 4 :

    https://youtu.be/ROiPyU2PGRM

    It may be more possible to physically detect the pluck of any string and with the pitch of any given string being a known value, simply send the midi data corresponding.
    This would in theory work faster than pitch detection and have the added benefit of polyphony.

    As for hardware to detect each string.... not sure, many possibilities. Many. Piezo sensors, infrared rangefinders, accelerometers, many possibilities most of which are probably bad ideas...(for lots of reasons starting with mutilation of the poor harp to hang sensors off of it. )

    The "Audio design tool" simply assists you in creating routing code, it doesn't create any functions.
    Thank you very much for your response.
    I had a good idea that the project was ambitious and that it would hardly be feasible (especially alone and with my current level of knowledge about Teensy).

    However, I would like to do some tests for the pleasure of learning.
    Would it be possible for you to make available a piece of code and a screenshot of the Audio Design Tool so that I can find out a little bit about how it all works?

    I've already developed the sensor system for the harp. I use infra-red to maintain the use of the harp in acoustics. The results seem to be correct...

    Best regards

  5. #5
    Junior Member
    Join Date
    Mar 2020
    Posts
    4
    Quote Originally Posted by oddson View Post
    yeah... from audio you are unlikely to get the results you are after. Polyphony would be somewhere between very tricky and nearly impossible.

    A MIDI harp with detectors on each string wouldn't be very difficult (to code) but might be to build especially if you want it to still work as an acoustic instrument.
    Thank you
    I'd still like to try to run some tests on a few strings to understand the process...
    The idea is indeed to keep the acoustic version of the harp and to add contactless sensors.

  6. #6
    Member
    Join Date
    Sep 2017
    Location
    Italia
    Posts
    87
    It could be possible to build a pickup for each string.
    This could even be something very simple, like a tiny LED reflex sensor on each string, because the pickup does not need to have exact acoustic qualities, but you could still get modulation information. If metal strings are used, even capacitive coupling or a homebrew tiny magnetic pickup for each string could be used.

    If you want something professional, then a piezo transducer for each string would be the way to go. That way you could get information on modulation as well, like how the string is played. For this it would be necessary to add some kind of bridge for each string to the neck. And what about the pedals, if the harp has some.

  7. #7
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,352
    For 'professional' version I'd think the piezo would be integrated into the eyelets on the soundboard. If the pedals are mostly 'digital' info (barring anything fancy like note-bending... is that a thing with harp playing?) then it's simple enough to have different offset maps for note values based on which pedals are engaged.

    It would be a massive build to do right. And you need to mess up a very expensive instrument.

    If you found one cheap enough that needed a lot of work anyway (broken soundboard??) then it might be feasible for a home project... but there would be considerable challenge in both the coding and, especially, the physical build.
    (And if you went to all this trouble you'd definitely want to sum all those analog signals into mono/stereo/quad type signals to play with in the analog world!)

    On the 'polyphonic pitch as monophonic audio to MIDI converter' issue; it is a very general problem and if there is a really good solution it should be commercially available. But I've not heard of one.

    Tracking and latency are always going to be a limitation because of the physical limitations of pitch detection -- even a single pitch. And having note-on events that are sent with the pitch info (per MIDI) means you normally wait not for the envelope detector to say 'note-on' but for the pitch-detector which must wait for a coherent fundamental to form (non-harmonic content during pluck on stringed instruments is very high and waiting for the fundamental to predominate means the note has been 'playing' for a while before the pitch is fully detectable).

    An equal temperament instrument with no pitch bend like piano and (i assume) harp will be simpler than a guitar but I think we're just not there yet.

    And for polyphonic you'd need to have an FFT output and then analyse to pick out the notes from the harmonics. The more accurate the FFT (smaller bins) the longer the latency and the more processing.
    All to say, not going to happen soon and it's not likely to be some-guy building it on Teensy (but you never know).

    Anyway... that's my amateur understanding of the issue.

  8. #8
    Member
    Join Date
    Sep 2017
    Location
    Italia
    Posts
    87
    To put the piezos to the soundboard is not exactly a good idea, because the soundboard is the thing that makes the sound and vibrates. This would trigger almost all piezos. Very hard to destinguish a weak note from crosstalk there.
    The usual harp has those bridge pegs (I don't know the proper english term) on the neck anyway, the ones that keep the string where it should be. There would be a chance to pick up the vibration of individual strings without too much of crosstalk.

    The pedals cam be used to pull notes for halftone, blue, etc. and are not necessarily digital, but normally analog, as they pull the strings via mechanics in the neck. So the position needs to be reported in analog mode.

    To use the strings as touch sensors and derive exact timings from them and the modulation out of an analog/microphone pickup signal via corresponding bandpasses or FFT is not practical, as harps usually have non-metal strings or at least these half-tone mechanics in the neck, that are made of conductive metal, electrically connecting all strings together. It would be a hell of work to separate these.

  9. #9
    Senior Member
    Join Date
    Mar 2013
    Posts
    131
    I have done research on various aspects of this challenge for a few years.

    Some companies did build piezo transducers (RMC, Lyon and Healy) for harps. They are fitted on the sound
    board so the string couples directly to them. They combined the sound (by putting them in series). For polyphony
    you will not want that. You need a charge amp for each one, i.e. 36 opamps.
    You end up getting a lot of crosstalk between strings so trigger detection is challenging.

    I experimented with a transducer fitted at the top of the harp where the peg turns the corner on its way to the tuner.
    It was doable but I couldnt afford to scale it x36.

    These days I would be tempted to throw DSP at the problem. If you know how your harp will be tuned, you can build
    a bank of comb filters centered at each pitch. Looking at the energy from them is a start to the trigger analysis. The principle of the comb filters is easy: https://www.dsprelated.com/freebooks...mb_Filter.html
    It is slightly complicated by the need to implement a fractional delay to tune them correctly. This approach avoids the latency and related challenges with FFTs.

    You will definitely get false triggers whatever you do because part of the charm of the harp sound is the way the strings excite each other and expand the resonances: it's basically a reverb. I tried making a harp that wouldn't do this communication between strings and
    and pretty much the only approach that worked was not very practical: concrete.

    e-mail me for further info, if you like (adrian@adrianfreed.com).

  10. #10
    Member
    Join Date
    Sep 2017
    Location
    Italia
    Posts
    87
    Yes, with brute DSP power it will be doable.
    At the end it would be easier to just learn how to play a keyboard, preferrably with polyphonic aftertouch and velocity. Then use a proper harp VST or whatever synth to make the sound...

  11. #11
    Senior Member
    Join Date
    Mar 2013
    Posts
    131
    Quote Originally Posted by radias View Post
    Yes, with brute DSP power it will be doable.
    At the end it would be easier to just learn how to play a keyboard, preferrably with polyphonic aftertouch and velocity. Then use a proper harp VST or whatever synth to make the sound...
    You can't get close to harp playing styles with a keyboard with polyphonic aftertouch and velocity. Also most VST and harp synthesis don't do proper tension-modulated pitch simulation which is key to the sound in many styles (e.g. Son Jarocho).
    Your suggestions are good if all you want is a basic controller that happens to use harp gestures, but you can't easily remove
    the sounds from the harp so presumably it is a hybrid or "augmented" instrument to aim for?

    https://www.youtube.com/watch?v=X2RNwasEdhk&t=75s

  12. #12
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,352
    Quote Originally Posted by adrianfreed View Post
    You will definitely get false triggers whatever you do because part of the charm of the harp sound is the way the strings excite each other and expand the resonances: it's basically a reverb. I tried making a harp that wouldn't do this communication between strings and
    and pretty much the only approach that worked was not very practical: concrete.
    But in the Harp as controller you only need to detect sudden attack as a note event. Sympathetic resonance should have an onset too slow to mimic a pluck. The signal should have transient qualities that you're able to detect and if an occasional note-on with low velocity slips through it's unlikely to sound bad as it will probably be an octave or fifth.

    And I suspect the amplitude from direct physical contact perpendicular to the tension on the string might make it very easy to tell plucks from resonance but it's the broadest of hunches and I'm not even sure a DIY eyelet piezo is feasible.

    But if there is some way to drill thru a disk without shorting it and put the eyelet through the disc (insulated but mechanically conducting) then the changes in tension on the string should produce large signals relative to the transverse waves in the soundboard itself. (Even broader hunch!)

    As for sensing from the other end, I would have thought it too complicated if the strings terminate differently based on the pedal position.

    And for pedals needing an analog of position: do they? If the positions are discrete they can be digital (if not merely Boolean) and it's only if you can bend notes that you'd need analog and then you also need to look at pitch bend messages if you expect to represent this as standard MIDI.

    In the tiny bit of research I did I did not see an acoustic instrument with MIDI. Most piezo systems uses a few pads because the soundboard does not convey the other notes particularly well and the only MIDI controllers I saw gave up on being acoustic instruments.

    I think it's possible but not really feasible because of cost, effort and difficulty coupled with a lack of any commercial potential.
    Last edited by oddson; 03-13-2020 at 04:44 AM.

  13. #13
    Junior Member
    Join Date
    Mar 2020
    Posts
    4
    Hi,
    I'm sorry I didn't post any news on the forum following your remarks, but with the current world situation, I don't really have my head in my projects anymore.
    I will resume all this a little later...
    In the meantime, take care of yourself and protect yourself until better days come...

    Michael

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •