Velocity-sensitive, contactless MIDI keyboard with polyphonic aftertouch for Teensy synth

[tomas]

Since there are so many synth projects on this forum, yet just a few talk about controllers for synths, I would like to introduce you to one of the projects I've been working on for some time... and I'm trying to motivate myself to finish it.

From the old times I got a musical keyboard made on an injection molding machine

The problem with it was that it only had mechanical contacts and only one contact per key, which provided only on/off readings, meaning without dynamics. Such an "organ" keyboard.
Besides, the contacts corroded over time and despite implementing various bounce-filtering algorithms, it wasn't possible to achieve adequate reliability during playing.

So, I thought about a modernization aimed at two things:
- Reliable key press detection
- Reading the velocity (force) of the key press.

The classic approach used in all keyboards produced by renowned manufacturers (Fatar - keyboards in most synthesizers from various companies, or Yamaha and Roland's own solutions) is membrane (rubber) switches with two switches per key. The velocity is calculated by measuring the time between the first and second row of switches making contact.

This is still a mechanical solution.

However, I thought that I am not limited by "standard" solutions and can do anything.

I was most interested in eliminating mechanical contacts, which are unreliable and characterized by noise (interference) during on/off switching.

At first, I thought about an optical solution (with optocouplers), but I quickly gave up.

Then I thought about using analog Hall sensors.

For those less familiar with electronics - an analog Hall sensor is an electronic component that converts the intensity of a magnetic field into voltage. When a magnet is placed near the Hall sensor, the voltage at the Hall sensor's output will change depending on the position of the magnet.

By using Hall sensors under each key, I could not only read the velocity between two fixed points but also across the entire length of the key's movement and also sense the position after pressing. Since the keys rest on a felt pad, the position measurement after pressing the key will correspond to the pressure on the key after pressing (aftertouch).

In this way, I simultaneously have both velocity sensitivity and polyphonic aftertouch.

It should be noted that polyphonic aftertouch is very rare in synthesizers due to cost.
Typically, synthesizers have only one sensor (force-sensitive resistor) for the entire keyboard, and thus aftertouch is mono.
One of the few synthesizers that had polyphonic aftertouch was the Yamaha CS-80 - Vangelis's favorite instrument, and thanks to the exceptional expressiveness of the keyboard with polyphonic aftertouch, the amazing soundtrack for [Blade Runner](https://www.youtube.com/watch?v=hhffOxGVGrY) was created.

In my solution, each key is read individually in full analog, which allows for the implementation of polyphonic aftertouch.

Conceptually, the matter is quite simple. Under each key, there is a neodymium magnet, which, when pressing the key, moves closer to the Hall sensor

The Hall sensors are connected to a 16x1 analog multiplexer (HC4067 chip).
Since my keyboard has 4 octaves, I need 4x12 sensors = 48, so 3 boards with multiplexers. The output of the 3 multiplexers will be connected to the analog inputs of the Teensy, which will handle the A/D conversion and will implement MIDI over USB.

This is how the keys with attached magnets and Hall sensors with the analog multiplexer look like

The schematic of one multiplexer board:

What is shown in the photos is done already.
I just need to connect all of this to the Teensy and program it.

I hope that sharing this project with you will motivate me to finish it soon

Stay tuned.

 

[eeeeeeeeefgsde]

This is such a cool project. I'm also in the business of refurbishing an old keyboard - in my case, giving a toy keyboard with rubber dome switches nicer-feeling mechanical switches. Still only on/off. However, I've realised using hall effect sensors would allow me to have aftertouch. Tempting!

Really curious to know if you made any more progress on this, and if so, how it went.

 

[tomas]

Thank you for showing interest ! This is appreciated. I hope to report some more progress this week.

 

[BillFM]

Hello Tomas.
The method of using magnets and Hall Effect sensors has been implemented by some people over at organforum.com. The people at this forum also use a lot of Teensy's (and Arduino's). You might want to check it out (search for Hall Effect), gather other people's experience. It's a pretty friendly crowd.

As I recall, here are some issues they ran into.
1) Adjacent magnet interference, false detection on adjacent keys
2) Usable range of Hall Effect output
3) Non-linearity of Hall Effect sensor
4) Magnetic drag on keys. Key press/release mechanical dynamics were affected by strong magnets.

Some solutions to the above issues were:
1) Use much weaker magnets
2) Install magnetic shielding between keys
3) Write software to characterize the sensor
4) Write software to compensate for sensor characteristics
5) Adjust spacing and limit travel between magnet and sensor.
6) Alternate magnetic poles. Every other magnet has its North pole facing upwards.
5) And in some cases, replace sensor with 2 magnetic reed switches. i.e., scrap Hall Effect method all together. Of course, this eliminates per key after-touch.

Many of the organforum projects using this method were installed on organ foot pedals. So, lots of room to make adjustments, add shielding, etc. A piano style keyboard doesn't afford this luxury.

So, when you observe some weird behavior, maybe it won't be too much of a surprise. The Hall Effect method definitely works. I suggest you start by sensing just 1 key and build from there. BTW, Yamaha GS-1 synth also has per key After-Touch.

 

[tomas]

I wasn't aware of organforum. Thanks for the pointer. Before I wrote first post, I already made experiments with few keys. I am using really small magnets that are specifically selected to minimize effects of magnet interference. Magnetic drag with the small magnets that I use is non-existing (I can feel absolutely no difference when playing).

With regards to supposed non-linearity:

Hall effect sensors are very linear. Actually they are pretty perfectly linear. Below is a characteristic of cheap SS49E analog Hall sensor

What is non-linear is relationship between field intensity and distance from magnet. It is inverse square law:

Field Intensity = Constant/ (Distance^2).

This can be easily accounted for in software. And inverse square law helps to diminish interference as field intensity drops quickly with the distance (in my keyboard the distance between sensor and the key is less than spacing of keys and the shape of magnetic field is focused centrally over the right sensor and the neighboring sensor field is weak, in my early experiments measured interference between keys was less than 3% )

Magnetic field intensity on the "side" of cylindrical magnet is much weaker than in the center hot spot:

Also the keys in my keyboard have pretty much small travel. I guess that people using organ keyboard may experience a lot more problems because their keyboards have longer travel.

 

[MarkT]

Magnetic field strength falls as the inverse-cube of the distance from a magnet, once far enough away for it to look like a dipole. To the best of our knowledge magnetic monopoles don't exist - if they did they would have an inverse-square law which is much more pervasive at distance.

 

[BillFM]

I wasn't aware of organforum. Thanks for the pointer. Before I wrote first post, I already made experiments with few keys. I am using really small magnets that are specifically selected to minimize effects of magnet interference. Magnetic drag with the small magnets that I use is non-existing (I can feel absolutely no difference when playing).

With regards to supposed non-linearity:

Hall effect sensors are very linear. Actually they are pretty perfectly linear. Below is a characteristic of cheap SS49E analog Hall sensor

View attachment 34866
What is non-linear is relationship between field intensity and distance from magnet. It is inverse square law:

Field Intensity = Constant/ (Distance^2).

This can be easily accounted for in software. And inverse square law helps to diminish interference as field intensity drops quickly with the distance (in my keyboard the distance between sensor and the key is less than spacing of keys and the shape of magnetic field is focused centrally over the right sensor and the neighboring sensor field is weak, in my early experiments measured interference between keys was less than 3% )

Magnetic field intensity on the "side" of cylindrical magnet is much weaker than in the center hot spot:

View attachment 34868

Also the keys in my keyboard have pretty much small travel. I guess that people using organ keyboard may experience a lot more problems because their keyboards have longer travel.

I'd say you've done some GREAT homework. Bravo!
Now, get coding and let us know how it goes.

 

[tomas]

Magnetic field strength falls as the inverse-cube of the distance from a magnet, once far enough away for it to look like a dipole.

That is even better. With (1/distance^3) interference between keys is much lower. And with very close distance to one of the poles (when key is pressed and one pole of the magnet nearly touches the sensor), I got more of 1/distance^2 characteristic which offers better handling of aftertouch.

 

[tomas]

Yesterday I wired the keyboard to the controller and found one tiny error in the printed circuit. After fixing it I wrote the program that simply reads
the Hall sensor values from all 48 sensors using three channel ADC and analog multiplexers on my custom boards.

Results are fantastic. I am getting positional sensing across entire range of key movement. And there is nearly no interference at all between keys as shown in the picture below. There are lines showing all 48 sensor values using 12 bit ADC (values 0..4095). As you can see the sensors are uneven in their "neutral" values, but this is to be expected as the data sheet specifies that neutral value output at 0 Gauss may vary from 2.25V to 2.75V and that is pretty much what I see on my end. This does not bother me much as I am more interested in "saturated" end of the measurement.

On the screen higher values are "up" (released) keys, and lower values are "down" (pressed) keys. I have pressed the "B" key and then pressed neighboring "C" key simultaneously, as you can see there is almost no interference between neighboring keys.

On the second screenshot you can see how precise is position sensing. First I press the one key slowly - see smooth downward curve reading the position of the key. Next I press another key (this time faster) and after a second I apply MORE pressure in "sinusoidal" motion but only to that SECOND key. See how pulsating pressure is read independently from the other key.

Overall I am super optimistic about the outcome as hardware performs very well.

I just need to address differences between sensors. I think that I would store min/max readouts of each individual sensors and scale its output accordingly to 100%. I also need to decide how fast I would scan the keyboard. Currently I am using 16kHz scan rate for each multiplexer channel which gives me 1000 of scans of entire keyboard, so maximum 1ms latency. Much better than any mechanical keyboard. I am also using hysteresis (lower note-on level than note-off level), to make sure triggering is very reliable.

Things to do:
- implement correction for differences between sensors
- measure velocity at the trigger point
- implement velocity at trigger point -> MIDI velocity mapping
- implement position after touch -> MIDI after touch mapping

 

[BillFM]

Amazing work!

Did you notice how after touch is affecting all other released keys?
It might be worthwhile (mostly out of interest) to plot only a few keys at a time to see how far away (plus or minus 10 keys?) after touch is reaching.

BTW, 1333 scans of each key per second, is used by one synth manufacturer. So, you're in the ball park.

 

[tomas]

Yes I noticed that. I am now thinking why it is so, but on the other hand it doesn't bother me that much, as I am using threshold to detect pressed key and currently I have set note-on threshold at level 2000, so it is far far away from any fluctuation that occurs on released keys.
The other thing is that the felt material that pressed keys land on is old and stiff. I am thinking of replacing it with new more elastic one, that would allow more aftertouch "travel". Currently the material is so stiff I need to actually press they keys quite hard to get substantial aftertouch response. I suspect that part of the released keys response to that pressure is due to some bending of the key bed when I press too hard
I am suspecting that there is a "residual" magnetic field from all the magnets that affects all released keys and when key bed is bending a millimeter or so you get that fluctuation on released keys.
With more elastic material I would not need to apply much pressure.
I may also check if there is fluctuation in voltage in the power delivered to the hall sensors.
So yes, this is something I will investigate further.
Thank you very much for showing interest!

 

[BillFM]

You could very well be on the right track about key bed bending during after touch pressure.

To check after touch influence on voltage rail, as an experiment, you could remove one of the Hall sensors and replace it with a 2 resistor voltage divider. Now plot after touch as you've done previously while pressing the key adjacent to the resistor'd key. If you observe after touch influence on the resistor'd key, then I'd say you have power rail/gnd disturbance. Play with the values to make the voltage nearer the upper rail, mid-rail, lower rail. I suppose resistor divider impedance will also come into play. Since the Hall sensor has a decent output drive, lower valued resistors might be called for. 1K Ohm to 10k Ohm range should be good to try.

To help equalize sensor output, you could bend the sensor legs a bit closer (or further) to the magnet. This might have an ill effect on sensor output range (or not).

Another experiment. For trying different soft materials (in place of felt), go to the hardware store and buy some O-rings used for plumbing faucets. It's a very soft rubber. The O-ring outside diameter will dictate how much force is needed for after touch. Example: a small O-ring that completely covers the key contact area will require the greatest force for after touch. Whereas a very large O-ring might only have contact along 1/2 its surface. The O-ring thickness will also contribute into this equation.

 

[SlowReplier]

Great project! I would like to make myself such a keyboard with 88 keys I look forward with excitement to the progress of integrating everything together with the microcontroller! Good luck!

 

[riban]

I am reallyinterested in how you get on. I was just posting about plans (well - aspirations) to do something just like this. I was inspired by posts in this thread then posted my ideas in this thread but whilst doing that I came across your post here. My inspiration was mostly from the ability to chose the onset trigger point to allow organ (very light) and synth (heavier) touch response but the points you make here about contact tarnishing, etc. were also in my mind. (I shudder to think what the contact rods are like in the keyboard I started building 40 years ago that has lived in my garage for a few of those decades!)

I am interested in the physical size of the magnets you used. I was looking at 8mm diameter, 2mm thick. You mention that you use "small" magnets to avoid adjacent key interference but I don't think you mention the size.

I was just about to order some magnets and was going to get both ferrite and NdFeB to see which worked better. You mention using the latter. Did you consider using ferrite which are supposed to be cheaper (but I don't see that), may be more envoronmentally friendly, are weaker and less suceptable to corrosion?

Your circuits don't show connection to the microcontroller. I assume you are multiplexing 16 sensors (as shown in your diagram) into each ADC pin of the microcontroller and attaining the 1kHz sampling rate mentioned. A Teensy has 10 ADC pins which would give access to 160 keys (2 pianos!!!) which is a good quantity of keys, pedals, etc.

The graphs show most of the dynamic range to be in the key throw with little within the aftertouch zone. Do you have any figures of how sensitive that zone is, i.e. what resolution of aftertouch are you able to detect?

I had thought of adding a softer foam / rubber bed to provide a larger aftertouch range. It could work well if the normal key travel hits the end stop / bed where there is sufficient resistance to feel like a normal keyboard then some extra motion with a little (but not excessive) extra pressure. There are MPE keyboards that do this really well where you don't even know there is another zone of motion but the second zone is actually quite long (not just the resistance of a rubber pad that traditional aftertouch offers).

The analogue Hall effect chip supports +/- signal based on the polarity of the magnetic field. Have you considered / investigated positioning the magnet to span its poles to increase resolution? Maybe its not required and just adds complexity.

It could be advantagous if you could plot (or indicate) the physical distance of key travel against the detected voltage.

Good luck with the project. I am looking at some unloved keyboards that might get some of this treatment soon...

[Edit] It looks like you have attached the magnets to metal brackets attached to each key. How did you fix the magnets (epoxy adhesive?) and do you think the metal brackets might affect the magnetic field, either now or in the future (possibly becoming more magnetised themself)?

 

[tomas]

Thank you for showing so much interest in my project! It is really appreciated!

To quickly answer some of your most important questions:

1. I am currently using neodymium magnets with diameter of 6 mm and thickness of 2 mm. No I did not consider other types of magnets (i.e ferrite).

2. Yes I am multiplexing 16 sensors into single ADC pin. For 48 keys I need 3 ADC lines on microcontroller (3 * 16 = 48). The design allows to connect any popular microcontroller including STM32F103 as well as Teensy 4.

3. Yes aftertouch zone is shallow and as I mentioned earier I am looking for replacement of old felt with something that would increase aftertouch zone

4. Yes Hall sensors that I am using SS49E are bipolar but I am using only "one side". To increase resolution I am powering hall sensors with 5V. The output of the sensors is (or rather should be) centered around 2.5V in this case. I am using only one side (the one that gives output voltage of 1V..2.5V). This is to ensure that I am not exceeding 3.3V.

5. I am not sure what you are suggesting with "positioning the magnet to span its poles"

6. I could increase keyboard scanning frequency, but frankly I don't feel any need to read keyboard faster than 1000 times per second, especially considering the fact is that it is already faster than old school MIDI. There is some propagation time that is required for the voltage at the output of the analog multiplexer to stabilize after multiplexer address has changed. I am not sure how I would systematically measure that. But I can try to come up with some ballpark figure. Some of the wires are half meter long. I think that 16kHz multiplexer switching frequency is probably very safe since it is 62 µs time per switch. These are CD74HC4067 analog multiplexers. According to data sheet they have "ON" resistance of less than 270Ω now assuming that flat cable has capacitance of approx 50pF per meter it would be 25pF for cable and add extra 25pF for everything "along the way" to ADC input. The time constant of this RC circuit would be 270 * 50 * 1e-12 = 13.5ns. Now to charge capacitor to 99.9% of source voltage requires approx. 22 time constants. So time required to charge ADC input line of the microcontroller would be around 300ns. That gives large safety margin when I am only switching every 62 µs

7. I haven't yet glued the magnets since I am experimenting with positioning them, so I want to have freedom to move them. Currently they are just kept in place by magnetic force (yes those metal brackets become a bit magnetized). Once I am happy with position of magnets I may fix them using glue/epoxy/or even double sided tape, I don't know yet.

 

[tomas]

OK, my calculations above are off by factor of 4.7 because I have 1kΩ resistor in series after multiplexer so actual time constant is 63ns and time required to charge ADC capacitor would be 1.4µs , but this does not change the fact that with 62 µs switching period I have quite large safety margin.

 

[riban]

positioning the magnet to span its poles

If a magnet is passed accross a detector so that its north pole is over the detector at one extreme and its south pole is over the detector at the other extreme you get full range from the sensor, doubling the range / resolution. You have since explained that you are using 5V supply but measuring 0..2.5V into a 3.3V device which makes sense. I recommend you provide some protection to the 3.3V input, e.g. zener diode to avoid inadvertent over-voltage.

I will order some 2x5mm magnets and start playing...

 

[tomas]

Yes I thought I needed extra protection, but then I checked 4067 analog multiplexer data sheet. https://www.ti.com/lit/ds/symlink/cd4067b.pdf
Every input is protected internally (see below) and since I am using Vdd=3.3V supply for the analog multiplexer, I seem to have that covered, especially considering the fact that I am supplying SS49E with something like 4.5V only (note bipolar transistor keys in the schematics) and at full swing SS49E outputs not more than 3.8V, so I am within 4067 specs.

 

[tomas]

There is one thing I have noticed that I think is the reason why Hall sensors aren't used in commercial musical keyboards design.
As it turned out during experiments, Hall sensors are susceptible to electromagnetic field (no surprise here really, after all they are magnetic field sensors ).
The ugly truth is that environment is polluted with EMF.
One night during experiments I switched on some LED lamp with magnifying lens. As it turned out the lamp was designed with poor impulse power supply that generated so much EMF that you can clearly read the EMF noise from sensor readouts. When lamp was close enough (less than half meter and connected to same power outlet as the computer), the EMF occasionally caused false triggering even if I had large hysteresis in the software.
I can imagine than on the scene when you are doing a gig, there must be tons of EMF that could cause malfunction of Hall-sensor based keyboards.
I suspect that the magnetic shielding that the other person was talking about, was not for key interference, but for shielding against external interference.
Solutions based on pressure sensitive resistors (like Yamaha CS80) do not seem to be sensitive to EMF (at least "on paper").

 

[riban]

The 4067 protection looks like a useful benefit.

Yes! EMF is going to be a big problem on a (electrically) noisy stage. It would be interesting to repeat your test with the noisy lamp and introduce some shielding to see if the issue is resolved / improved, e.g. connect some kitchen foil to ground and place over the key mechanism (covering the sensors and magnets).

 

[sbfreddie]

Tomas:
Aluminum foil will not shield anything from a magnetic field. What you need for magnetic shielding is mu-metal. It comes as Conetic, and Netic.
One shields sensitive circuits, the other shields strong magnetic field emitters.
EMF can come from many different sources, and many different frequencies. The EMF from magnetic devices, such as transformers, electric motors, fluorescent lights with magnetic ballast, DC to DC converters, etc. can produce strong H-fields surrounding them, which can radiate a 50Hz, 60Hz, or whatever frequency the DC to DC converter is operating at, H-field everywhere. These sources have been causing all kinds of havoc in sensitive electronic devices for a very long time, and most engineers that I have had contact with over the years have never been aware of this problem, or how to mitigate them.

Regards,
Ed

 

[riban]

I was suggesting a temporary experiment to test the effect of shielding from EMI, not shielding from extraneous magnetic fields. If you put a big / strong magnet on your keyboard it will give interesting (maybe predictable) results. If you turn on some high voltage arcing equipment (or similar) you may get unpredictable electromagnetic interference over fairly large distance. Similarly if you have a device that is creating radio waves it could cause interference. A good earth / ground bonded to some aluminium foil can provide a degree of shielding of EMI which could indicate how best to implement a more robust shielding solution. This is different to protecting against magnetic fields that can have a profound effect but may be mitigated in other ways, like physical distance.

Both need to be considered and mitigated appropriately.

 

[sbfreddie]

Riban:
Agreed!

Regards,
Ed

 

[sbfreddie]

Tomas:
Here is another test for sensitivity: Setup a large subwoofer speaker, or a large guitar amp speaker, or any amplified music device with large speakers close by with the volume cranked up and playing a serious baseline, and see what you see on the Hall effect sensors, you may be surprised.

Regards,
Ed

 

[tomas]

Thank you all for comments.
I am not very surprised that Hall sensors are sensitive to magnetic fields - that's what they are made for. I am also not surprised that speakers produce magnetic fields, that's their principle of operation.

With that in mind the purpose of this project is having keyboard usable in a controlled, relatively EMF-quiet, studio environment, rather than power station or stadium gig.