Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 13 of 13

Thread: Project Cyborg Ears - wearable 'DSP hearing experience' platform

  1. #1

    Project Cyborg Ears - wearable 'DSP hearing experience' platform

    Hello all,
    We would like to present you an extensive audio hardware project that we are currently working on, and some of the new ground we've explored relating to teensy audio capabilities.

    Our goal
    We aim to augment 'normal' human hearing with digitally processed realtime audio recordings.

    Concept: we have an audio DSP, audio CODEC and headphone amplifier chip that
    plays the audio captured by a pair of small microphones to regular off the shelf in-ear headphones.
    Unlike other designs, the microphones are not statically attached to the headphones, more on this in a later post.

    Who we are
    Chris & Johannes, two Master of Science students from Germany.

    Our implementation
    We use the teensy 3.1 microcontroller and the teensy audio shield which contains the SGTL5000 audio chip and a limited DSP.
    Recording is done by two tiny 24-bit ICS-43432 MEMS microphone on breakout boards (Overview, datasheet) that output sound to a single stereo I2S audio line.

    Top view on wearable experimental configuration of:
    • teensy
    • audio shield
    • special ICS-43432 microphone pcbs
    • custom black-blue audio data cable
    • headphones
    • usb-battery

    Click image for larger version. 

Name:	cyborgears_prototype_conceptview_small.jpg 
Views:	314 
Size:	61.8 KB 
ID:	4571

    This picture shows a close up for size comparison:
    Click image for larger version. 

Name:	mockup_headphone_mic_1_small.JPG 
Views:	515 
Size:	382.5 KB 
ID:	4572

    Motivations leading to this project
    Hearing aids have been socially accepted 'augmented hearing' devices for a long time, but:
    • What benefits could "digitally modified" hearing bring to people without hearing disablities?
    • Can our senses be improved or altered in a way that is useful or pleasurable to an individual by such a system?
    • Within which limits can this be used to protect the regular hearing system, e.g. at a concert?
    • How does it FEEL to actually hear things from a point outside of your body?
    • What emotions will be brought up by altering the way the world sounds to us?

    Those are just a few questions that we are able to explore with our system.

    Simulated hearing experience
    Although we were not completely aware of this fact at the beginning of our project, our area of research is closely linked to the topic of Binaural recordings, where stereo sound recording is done at the ears of real or synthetic heads, with the goal to capture the authentic way the sound would reach a spectator's ears.
    In contrast to this, the ability to record is not as important for us. While our platform could later be modified to record the sounds captured by our microphones to an SD card or computer, our primary goal is focused on the realtime experience. (This is partly due to our planned signal path. Also, there are many commercial products already aimed at binaural recording)

    That said, binaural recordings are a good way to demonstrate the effects of an altered hearing experience and get into the topic of how immersive sound can be.
    If you would like to try this, put some headphones on and listen to this binaural example from wikimedia.
    Imagine yourself actually crumbling this paper in real life, but hearing it with swapped "ear"-sides (left ear recordings go to the right, and vice versa) - swapping sides is one of the easiest and at the same time most impressive ways to alter our hearing. It's also one of many effects that do not actually improve your hearing, but invites you to explore things we take for granted in our daily life.

    Available DSP & mixer capablilites
    As the SGTL5000 DSP was (probably) designed for media devices like smartphones, it's DSP features are very limited, and only include
    • amplification (multiple stages)
    • limiting sound loudness
    • sound compressor
    • proprietary bass enhance w/ cutoff filter
    • equalizer

    This means that many cool features that one can think of, e.g. hearing ultrasound or advanced noise cancellation, are not possible on this platform.
    As far as we know, the chip is also unable to do mixing of the microphone audio with sound from the line-in input channel, although there might be interesting application where it is enough to switch between them (listening to music on the train in one minute, talking to the train conductor in the next)

    Implementation story, part 1
    Achieving the working system we have now was not so straightforward:
    As you may know, the I2S block of the ARM Cortex M4 used in the teensy 3.0/3.1 has a separate I2S stereo receive and a I2S stereo transmit channel, so our initial plan was based on receiving the microphone data on the teensy with the regular audio library created by Paul, doing some sound processing/mixing, and then transmitting the results to the audio shield via the I2S TX. There, the DSP could optionally apply some changes and then finally deliver the audio to the in-ear headphones.
    This way, we'ld limit ourselves to a one way street of mics -> teensy processing -> teensy shield processing -> headphones, but could mix in e.g. synthesized sounds or any other teensy DSP function.

    Unfortunately, the microphones provide fixed 24-bit audio data, while the internal ARM Cortex M4 DSP instructions are limited to 16 bit on a hardware level. For this reason, Paul had previously decided to hard-code his audio processing library for 16-bit sound data only (!), and we were unable to use or adapt it due to the low-level DSP instructions that are used throughout the library, as we decided against downconverting the audio to 16 bit for reasons of complexity, speed and audio quality.

    Our current solution is to directly connect the microphones to the teensy audio shields I2S RX channel, which reduces the teensy to I2C control in theory.
    ('Passively' receiving the same microphone data on the teensy I2S RX is still an option, but not used. The actual I2S configuration is basically a hack and more complicated as explained here, but more on that later.)

    To achieve direct microphone->audio shield input, we've successfully switched the SGTL5000 CODEC configuration to accept stereo 24 bit of audio data (in 32-bit I2S dataframes) at 48kHz sampling rate. (Choosing 48kHz over the regular 44.1kHz was a voluntary step and not necessary for the signal path)

    [Update]
    To clarify: our current signal path looks like this:
    2x I2S mics -> teensy shield w/ SGTL5000 -> stereo headphones
    ##

    So much for now. Please note: we're quite busy developing and might not be able to reply in time.
    Last edited by CyborgEars; 07-01-2015 at 04:57 PM. Reason: Re-uploaded images.

  2. #2
    [reserved post]

  3. #3
    [reserved post2]

  4. #4
    So if i am not mistaken you can manipulate your data streams only trought the SGTL and the teensy act only as control of the audio board ? With you hack to the SGTL codec are you able to mix to 2 imputs subtracting them and acting as a noise cancelling monoaural device ?
    Your project if really cool ! Thanks .

  5. #5
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,576
    I'll look forward to hearing how this project goes.

    Are you using the Teensy Audio Library, or rolling your own signal processing software infrastructure?

  6. #6
    @riodda: Yes, that is correct, the SGTL5000 is our 'only' means to manipulate the audio data in this signal flow configuration.

    Regarding mixing and 'real' DSP work:
    The SGTL DSP is limited to enhancing a single stereo audio stream with the audio blocks I've listed before, which don't provide the necessary functionality.
    Look at page 2 of the datasheet for a better overview. Note that the "audio switch" pretty much is a one-input one-output switch, instead of a mixer component as one might expect - or need for eg. noise substraction.
    EDIT: This is not 100% correct, see next post

    Also, most effects are performed symmetrically on both the left- and right- audio channels, at least on the way they're exposed by the current teensy SGTL5000 control code (see below). Volume can e.g. be regulated independently, although our current code does not.

    @Paul:
    As mentioned in the first post, we were unable to use (or adapt) your library for 24 bit without completely dismantling it or giving it a complete overhaul,
    and as the real benefits looked slim for our application we completely ditched it for the time beeing (mixing in either synthesized sounds,
    recorded wav files or audio grabbed via the ADCs would've been nice, but wasn't a must-have).
    Also, when we started in April, whollender's I2S code wasn't available yet.
    Baking flexible 24-bit configuration settings into the official audio library, even with very limited "DSP functions" support for 24 bit, would be great and keep development like ours or those of whollender's 24 bit ADC/DAC boards nearer to the 'mainline' branch.

    At the moment, we jumpstart the I2S with a modified version of https://github.com/nodae/teensy_i2s_experimental and then use control_SGTL5000.c, which we also modified to run the SGTL5000 in 24 bit depth mode (+48kHz sampling rate, but that was optional).
    This abstraction class is essential for controlling the 'DSP', and it's existence was one of the reasons we didn't opt for custom-build boards.
    By the way, we've found some issues along the way (e.g. confusing documentation of input variables, hardcoded 44.1kHz stuff, ..) which I'll go into detail in separate bug/improvement-threads for better readability.

    As always: thanks for making such cool open source stuff!
    Last edited by CyborgEars; 07-16-2015 at 03:53 PM.

  7. #7
    Exciting news: I've just realized that I was wrong about the claimed inability to mix ADC (=LINE-IN) with the main I2S_IN audio.
    While the audio switch in general is just a simple switch as mentioned, the SGTL5000 makers have also included an component to the DAP (digital audio processor) that we overlooked: the Dual Input Mixer.
    This allows us to mix the "normal" (MAIN) channel with an "additional" (MIX) component.
    AFAIK this is only additive mixing and controllable from 0-200% nominal volume of both streams.

    I just coded a quick POC into our current prototype and it works - I can hear the environment via the microphones while listening to music!
    (If someone wants to dive into this, look at CHIP_ANA_ADC_CTRL, CHIP_SSS_CTRL, DAP_CONTROL, DAP_MIX_CHAN & DAP_MAIN_CHAN)
    Last edited by CyborgEars; 07-02-2015 at 11:47 AM.

  8. #8
    Senior Member
    Join Date
    Oct 2015
    Location
    Vermont, USA
    Posts
    190
    Cool project. Is there a project webpage that might show more results since this last post in 2015?

    Also, for those not interested in making their own microphone-enabled earbuds, Roland sells such products: The Roland CS-10. https://www.roland.com/us/products/cs-10em/. Each earbud has a typical inward-facing speaker and and outward facing microphone.

    Click image for larger version. 

Name:	cs-10em_angle_main.jpg 
Views:	99 
Size:	8.2 KB 
ID:	8520

    Here are their advertising bullets:
    * Compact, high-quality stereo condenser microphones built into earphones
    * Powered by the recorder (operating voltage is 2V to 10V; also works with other products)
    * All-in-one design lets you monitor while recording
    * Sophisticated enclosure design reduces feedback

    Here are their specs:

    MICROPHONE
    Element: Electret Condenser
    Polar Pattern: Omnidirectional
    Frequency Response: 20 Hz to 20,000 Hz
    Open Circuit Sensitivity: -40 dB 1V/Pa
    S/N: Greater than 60 dB
    Impedance: 2.2 kΩ
    Power Requirements: 2 V to 10 V

    EARPHONE
    Type: Dynamic
    Driver: 12.5mm
    Sensitivity: 102 dB/1mW, 500 Hz
    requency Response: 15 Hz to 22,000 Hz
    Impedance: 16 Ω

    I have a pair. Note that these use traditional analog electret microphones, not the digital I2S mics shown in the original post. I've used these Roland earbuds/mics with the Teensy Audio board (via the MIC input). They work fine, though there is only one MIC input channel on the Teensy Audio board, so you only get Mono, not stereo. If you add in one's own little mic pre-amp, you could use the Teensy Audio line-input and be done!

    Chip


    I've used them with the MIC-in on the Teensy Audio board. It works fine.
    Last edited by chipaudette; 10-21-2016 at 02:30 AM.

  9. #9
    @chipaudette: as the teensy's ARM chip is permanently handicapped in the 24-bit audio department as it's DSP instructions only work on 16 bit samples, audio library support for 24 bit is notoriously lacking, too. Consequently, there was the need for intensive low-level work on basic I2S and CODEC configuration as well as many compromises (namely, not using the teensy audio chain at all, and having only rudimentary recording support), which had let us to switch to dedicated, large DSP audio boards a while ago, bringing it's own set of problems and are much less portable than teensy + shield + power source.

    Through a lack of time and shifting focus on other ideas, this one has been largely on hold for the last months.

    Regarding the Roland CS-10: this is a very interesting product that solves a lot of the underlying mechanical and electrical issues of our current path, namely placing on-chip microphones in an isolated fashion on commercial in-ear plugs and routing their high-speed bitstreams down well-shielded, flexible, yet tiny and robust cables. The obvious challenges with the CS-10 are the signal degradation through electrical interference and crosstalk (both at the plugs and between the unbalanced mic / stereo lines), which seem to be handled well according to the reviews I've read. The teensy audio shield, as many other portable consumer devices, is however only able to capture a mono mic source.
    As replacing your normal hearing with a playback of a live recording is quite demanding in terms of sound quality, we weren't looking forward to roll our own amp/ADC/DSP/DAC/headphone-amp solution, and most affordable microcontroller-based solutions seemed lacking. From initial tests and using the line-in in combination with regular audio sources, going this way with the SGTL5000 didn't look like a high-quality option. There might be commercial field recorders with the necessary non-delayed, DSP-enabled live monitoring headphone output that do this job out of the box while delivering outstanding audio quality - and I'ld like to know how well they work out at this implementation. Controlling their settings from a microcontroller (e.g. to switch current playback from line in to mics when on the train) for some kind of automation might prove challenging, however.
    Last edited by CyborgEars; 11-11-2016 at 03:10 PM.

  10. #10
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,576
    Quote Originally Posted by CyborgEars View Post
    Baking flexible 24-bit configuration settings into the official audio library, even with very limited "DSP functions" support for 24 bit, would be great and keep development like ours or those of whollender's 24 bit ADC/DAC boards nearer to the 'mainline' branch.
    The reality of limited hours in every day means I have to prioritize features. As you can probably imagine, I have an incredibly long list of requested features....

  11. #11
    Quote Originally Posted by PaulStoffregen View Post
    The reality of limited hours in every day means I have to prioritize features. As you can probably imagine, I have an incredibly long list of requested features....
    And you're doing a great job at it, not to mention the new hardware you manage to put in our hands every year

    The above comment of mine is a year old and represents the (yet unfulfilled) idea to make certain key elements (like the SGTL5000 control code) less dependent on fixed 16 bit 44.1kHz data, making them usable in 24-bit projects without the need of semi-private forks like ours that get left behind eventually. If you're interested, I can redo my old and apparently forgotten pull request #118 in a cleaner fashion against the current master branch, starting with cosmetic/documentation issues, but then moving up to exposing useful DSP functions and other improvements found along the way.

  12. #12
    Senior Member onehorse's Avatar
    Join Date
    Apr 2014
    Location
    Danville, California
    Posts
    921
    I will just mention that the STM32L4 is a viable option for sophisticated audio processing but some API development will, of course, have to be done. It has two SAI (I2S ports) engines with a wide array of rates and formats available, and it is likely much of what you want to do can be accomplished without a CODEC, especially if you keep using the ICS43432 microphones..

  13. #13
    Senior Member
    Join Date
    Oct 2015
    Location
    Vermont, USA
    Posts
    190
    I just pieced together a Teensy-based system for real-time audio processing involving many of the same elements discussed here. My goal was to make a hearing aid, but the exact same hardware could be used to do any sort of real-time audio mangling, as was the goal of the Cyborg Ears discussed above.

    Click image for larger version. 

Name:	TeensyHearingAidLead2.png 
Views:	110 
Size:	112.2 KB 
ID:	8852

    Thread here: https://forum.pjrc.com/threads/40115-Teensy-Hearing-Aid

    Chip

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •