Open Sound Control (OSC) Teensy Audio Library Implementation

[JayShoe]

Hello,

I'm curious of the feasibility and interest of controlling the Teensy Audio Library with Open Sound Control and the feasibility of implementing OSC as a baked in function of the Teensy Audio Library. I'm considering work on this, and want to know if anyone else had any useful input.

The goal is simple. For each audio object, write a helper object that automagically creates an OSC control address. For example, the "Mixer" in the audio library.



That object has a name "Mixer1" and has a public function for "Gain" with options for channel and level. What if this proposed solution would automatically create an OSC address of /mixer1/ch1/gain and /mixer1/ch2/level? Or as another example, the Sine function could be as follows.


OSC address:

• /sine1/amplitude
• /sine1/frequency
• /sine1/phase

If this were done on for each audio object, using a library, wouldn't that make creating a custom device with the Teensy Audio Library more easily accomplished? I haven't yet implemented OSC in any of my projects, but I did for MIDI. Unfortunately MIDI doesn't seem like the ideal way to communicate with the Audio Library because I had to write if statements for each MIDI channel I wanted to work with.

This example is for MIDI and how I monitor two sliders on my MIDI interface designed with TouchMIDI.

// TEENSY MIXER CONTROL GAINS
// AUX 1 
       if(channel == 3 && data1 == 51)
       {
         // Channel Volume Control
          volume3 = (float)data2 / 127; // save the volume
        // Calculate the total Volume
          i = volume3 * (1 + gain3);  Serial.println(i);
        // Set the volume
          mixerLeft.gain(1, i);
       }
       if(channel == 3 && data1 == 53)
       {
         // Channel Gain Control
          gain3 = (float)data2 / 127; // save the gain
        // Calculate the total Volume
          i = volume3 * (1 + gain3);  Serial.println(i);
        // Set the volume
          mixerLeft.gain(1, i);
        }
// AUX 2
       if(channel == 4 && data1 == 51)
       {
        // Channel Volume Control
          volume4 = (float)data2 / 127; // save the volume
        // Calculate the total Volume
          i = volume4 * (1 + gain4);  Serial.println(i);
        // Set the volume
          mixerRight.gain(1, i);
       }
       if(channel == 4 && data1 == 53)
       {
         // Channel Gain Control
          gain4 = (float)data2 / 127; // save the gain
        // Calculate the total Volume
          i = volume4 * (1 + gain4);  Serial.println(i);
        // Set the volume
          mixerRight.gain(1, i);
        }

This code was a bit tedious to write and understand, and it required the creation of a MIDI mapping table somewhere to keep track of all the controls. The HTML of TouchMIDI has corresponding code as such.

<!-- Aux 1 -->
<div class="column">
<div class="text" style="position: absolute">&nbsp Aux In</div>
<div class="encoder" label="Digital Gain" midicc="4, 53" colour="#FB1CF3" storeid="s1m1"></div>
<div class="slider" label="Volume" midicc="4, 51" colour="#14E528" storeid="s2m1"></div>
</div>

The reason this is better in OSC is because of the way OSC addresses work. We already have a name for each audio object, and we know what functions each object expects and accepts. So when going to create your controller elsewhere you have a clear understanding of what address the control you are seeking is located. So it seems to me that this would be a really nifty and feasible solution.

If the OSC addresses were automatically mapped out, this would make the OSC control happen pretty much automatically. It would standardize controlling the library with OSC, if you will. Right now I don't know of any examples that actually show how to control the Teensy with OSC. Does anyone know of any shared projects that show a full example, and maybe a corresponding control surface written for the desktop or browser? The CNMAT/OSC has some examples to send and receive control, but it's a little foggy for me to truly grasp.

Does anyone know how I would go about creating something like this? I envision a separate library (hopefully) so we didn't need to push into the audio library directly (creating more work and overhead on the library itself).

Any thoughts? Any interest?

Jay

[manicksan]

I'm interested

To make this possible you have to create a complete list of all functions for every AudioStream object in the whole "Audio Library" saved as a string array,
so that every function can be called by name.

but fear not, as I have made functionality that extracts that information from the "Design Tool" embedded documentation
it's used by the "auto complete" functionality for the code editor.

And now I made a simple function that takes every object name
and go through every help to extract all the functions mentioned there.

you can check it out here
https://manicken.github.io/
When opened go for the settings tab (right side)
then "Development Tests"-"Export complete function list"

note. it will not show functions that are not mentioned in the help.

Hope it will be a start
The output format can easily be changed,
example the whole lockup table can be autogenerated.

Also my new design tool can use designs by classes as well,
then a object address will be "className"/mixer1/amplitude
but that will be taken care of by the "Design Tool"

what is needed then is some code generated by the "Design Tool" that maps all generated objects to the OSC table.
simple example:

// note the type numbers are taken from the "generated" list above

#define OSC_TYPE_AudioSynthWaveform 47
#define OSC_TYPE_AudioMixer4 34
#define OSC_TYPE_AudioOutputI2S 14


void decode_osc_funcs_AudioSynthWaveform(AudioStream *osc_obj, const char *func_name, const char *func_value) {
    // pseudo code (don't know if it works) specially the string compare don't work like this
    float val = std:stod(func_value);
    AudioSynthWaveform* aswf =  dynamic_cast<AudioSynthWaveform*>(osc_obj);
    if (func_name == "amplitude")
        aswf->amplitude(val);
    else if (func_name == "frequency")
        aswf->frequency(val);
    else if (func_name == "offset")
        aswf->offset(val);
    else if (func_name == "phase")
        aswf->phase(val);
    // ... and so on
}

// same as above but for AudioMixer4 instead


AudioStream *asObjs[4];
int osc_types[4]; // used together with osc_names to determite which decode func to call
const char *osc_names[4] = { "wf0", "wf1", "mixer4", "i2s" };

AudioSynthWaveform               wf0; 
AudioSynthWaveform               wf1; 
AudioMixer4                            mixer4;
AudioOutputI2S                       i2s; 

void mapToOSC() {
    int asObj_index = 0;
    asObjs[asObj_index] = wf0;
    osc_types[asObj_index++] = OSC_TYPE_AudioSynthWaveform;
    asObjs[asObj_index] = wf1;
    osc_types[asObj_index++] = OSC_TYPE_AudioSynthWaveform;
    asObjs[asObj_index] = mixer4;
    osc_types[asObj_index++] = OSC_TYPE_AudioMixer4;
    asObjs[asObj_index] = i2s;
    osc_types[asObj_index++] = OSC_TYPE_AudioOutputI2S;
}
void decode_osc(const char *type_name, const char *func_name, const char *func_value) {
     // pseudo code
     for (int i = 0; i < 4; i++) {
         if (type_name == osc_names[i]) {
             if (osc_types[i] == OSC_TYPE_AudioSynthWaveform) 
                 decode_osc_funcs_AudioSynthWaveform(asObjs[i], func_name, func_value);
             else if (osc_types[i] == OSC_TYPE_AudioMixer4) 
                 //decode_osc_funcs_AudioMixer4 (asObjs[i], func_name, func_value);
             // ... and so on
         }
     }
}


void setup() {
    mapToOSC();
}

it's a mess but it should work in theory

I think it's best to autogenerate all decoding functions,
if we can just get one type working then it can be applied on all the others (by auto generation in the "Design Tool")

The CNMAT/OSC example don't care about function names at all, it's just the bare message handler.

[manicksan]

it should be nice if it could somehow be implemented together with
https://forum.pjrc.com/threads/66840...ffort-going-on

[h4yn0nnym0u5e]

it should be nice if it could somehow be implemented together with
https://forum.pjrc.com/threads/66840...ffort-going-on

Definitely looks interesting. At first glance it appears that an OSC class library could use AudioStream objects as its base class, and thus be made to work with static or dynamic audio objects. That would avoid the (new) burden of adding the OSC interface to every existing audio object, and mandating its implementation for every new one.

Auto-generation of the OSC classes would be great, though it would definitely require good discipline on the part of the AudioStream object's documentor (better than is currently apparent, I think!).

Cheers

Jonathan

[manicksan]

I have checked now
dynamic_cast cannot be used as it compiles with the -fno-rtti flag
which means that Run-time information is not included in the "compile"
and therefore the dynamic_cast cannot work.

but I have removed the -fno-rtti flag in boards.txt
and the compilation works.

Do anyone know the difference by using the flag and not?,
do it take more program memory/RAM?

I have also tested it on a Teensy 4.1
I use this code (based on code from https://www.geeksforgeeks.org/dynamic-_cast-in-cpp/):

// Base class declaration
class Base {
  public:
    void print()
    {
        Serial.println("Base");
    }
    virtual ~Base() {} // this enables Polymorphism
};
  
// Derived Class 1 declaration
class Derived1 : public Base {
  public:
    void print()
    {
        Serial.println("Derived1");
    }
};
  
// Derived class 2 declaration
class Derived2 : public Base {
  public:
    void print()
    {
        Serial.println("Derived2");
    }
};

void setup() {
    Serial.begin(9600);
// put your main code here, to run repeatedly:
// put your setup code here, to run once:
    Derived1 d1;
    Derived2 d2;
    // Base class pointer hold Derived1
    // class object
    Base* bp1 = dynamic_cast<Base*>(&d1);
    bp1->print();
    // Dynamic casting
    Derived1* dp1 = dynamic_cast<Derived1*>(bp1);
    if (dp1 == nullptr) {
        Serial.println("null");
    }
    else {
        Serial.println("not null");
        dp1->print();
    }
    
    Base* bp2 = dynamic_cast<Base*>(&d2);
    bp2->print();
    // Dynamic casting 2
    Derived2* dp2 = dynamic_cast<Derived2*>(bp2);
    if (dp2 == nullptr) {
        Serial.println("null");
    }
    else {
        Serial.println("not null");
        dp2->print();
    }
}

void loop() {
  
}

the output is:

Base
not null
Derived1
Base
not null
Derived2

which looks promising

[manicksan]

found the -fno-rtti flag reason

https://forum.pjrc.com/threads/41364...p-dynamic_cast

but even if it generates a lot of bloat,
on the Teensy 4.x there are now more possibilities with more program memory and much higher speed.
maybe we can have it as a user-setting in the menu (for teensy 4.x/MM)?

[h4yn0nnym0u5e]

Do we really need dynamic casting? So long as you don't cast to an incorrect class (one not in the hierarchy) it would appear that static casting would work as well for us, and avoid the bloat and non-standard compile-time options. But I could easily be missing something, I'm pretty unfamiliar with C++.

I found this OSC class library which looks as if it might be helpful; mentions Teensy and has had reasonably recent updates (compared to other options...).

[manicksan]

It works with the -fno-rtti flag and by using static_cast instead of dynamic_cast
and the code was smaller
15948 bytes vs 23556 bytes (without the flag)

while (without the flag)
I also did try using the typeid(d1).name() which actually returned the full name: 8Derived1

[adrianfreed]

This is a good idea and welcome on the Teensy from my perspective.

It's actually in the spirit of the first OSC implementations where we expected the name space to be synthesized
automatically from the signal flow graph of the patching language. I did this in a simple C-based system
(called HTM) with the first implementation of OSC.

[JayShoe]

Sorry for not responding sooner, I have been following this thread with much interest.

manicksan, your work on the Audio Tool is awesome! I have also been paying some attention to that project but I did not realize how far you've pushed the limits with it. I noticed that there are midi sliders and everything in there! I really need to take a better look. After seeing what you are doing with the audio tool, I can understand why this concept struck a chord with you. The audio tool could basically integrate the OSC control into the interface. So when you drop a mixer in, the corresponding OSC control could be automatically placed onto a control UI! That would be sweet! I really appreciate your notes, and I'm trying to read and understand your code. This is a learning experience for me - I'm always trying to learn more and your code is helpful to review (impressive).

adrianfreed, thank you for your contribution to the Arduino OSC library! It's nice to see you here on this thread. Do you have any simple examples that would show both sides of controlling an audio object in the audio library? The CNMAT examples seem to be relatively high level. It would be great if we had an example that completed a working example of changing a volume on a mixer, for example (including the recommended transport layer).

I'm not sure what the proposed control surface might be. Of course there is TouchOSC and others. It could also be nice to have an example of two teensies one for the audio processing and one as an OSC controller in hardware (what transport layer?). Or maybe this could be one Teensy, or maybe that's not necessary because you could control TAL directly at that point... But there is also javascript library for OSC control inside the browser here that I think would be worthwhile to review for this concept/project. https://github.com/colinbdclark/osc.js/

[h4yn0nnym0u5e]

I too have been following this with interest ... as @manicksan says, integration with my Dynamic Audio Objects (as a layer on top) looks promising.

I was thinking that it would be possible to use the Web Serial API along with the GUI to send OSC messages to place, rename, remove, connect, disconnect and control audio objects live. The immediate downside is that it only operates in a secure context. However, I had a bit of a play and found some instructions on setting up a secure local host using Python 3. I had to change the Python code a bit, ending up with

# Tested with Python 3.9.7 on Windows 10
#
# Create key.pem and cert.pem using
#  openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365

from http.server import HTTPServer, SimpleHTTPRequestHandler
import ssl

ssl_context = ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH)
ssl_context.load_cert_chain(certfile=r'path/to/cert.pem', 
                            keyfile=r'path/to/key.pem',
                            password='NotVerySecure')


httpd = HTTPServer(('localhost', 4443), SimpleHTTPRequestHandler)

httpd.socket = ssl_context.wrap_socket (httpd.socket,
                                        server_side=True)

httpd.serve_forever()

If you run this from the folder with the GUI in it, and point your browser at https://localhost:4443/index.html, you get the Audio Designer as usual, but it should now be possible to put Web Serial functionality in. I haven't tested that yet. Need to think a bit about the address patterns and methods to give a proper expandable ecosystem, which ideally works with both static and dynamic audio objects.

Cheers

Jonathan

[h4yn0nnym0u5e]

Anyone got thoughts on what should happen with the many "get information" functions provided by various AudioStream objects? For example AudioAnalyzeRMS::available().

[JayShoe]

Anyone got thoughts on what should happen with the many "get information" functions provided by various AudioStream objects? For example AudioAnalyzeRMS::available().

Does it make sense to have the addresses categorized?

Also my new design tool can use designs by classes as well,
then a object address will be "className"/mixer1/amplitude
but that will be taken care of by the "Design Tool"

aka:
category/className/object/control
eg:
control/className/mixer1/ch0
info/className/AudioAnalyzeRMS

Putting the category in there would make it clean. Would that make any sense?

[h4yn0nnym0u5e]

I was thinking that typically the className would be unnecessary for most uses: if your mixer1 is an instance of the AudioMixer4 class, the OSC address doesn't need the className to refer to it. However, for static functions then it would potentially be useful to have access via the className, which could take the place of the instance name.

For maximum flexibility, e.g. if OSC messages are broadcast via a network, it seems sensible to allow for a "node name" as the first address element. Within a node we might have several OSC-aware clients in addition to the audio engine, e.g. MIDI routing or front panel control client, so another level of address internally is probably worth putting in.

Say we have an AudioMixer4 called mixer1, and a MIDI router function: we could then address the former with an OSC message like /teensy1/audio/mixer1/gain,if<0><0.5> which results in a call to mixer1.gain(0,0.5). Then conceptually we might have /teensy1/midi/CCroute,ifs<32><0.0078125></teensy1/audio/mixer1/gain(0,v)> which tells the MIDI router to scale CC32 from 0-127 to 0-1.0, and send the value to mixer1 channel 0's gain. Need to think how that works for multi-parameter functions! Also, this supposes the use of OSC internally - not sure how efficient that would be.

Using dynamic audio objects, we can have /teensy1/audio/create,ss<AudioMixer4><mixer1> to create the mixer on the fly, /teensy1/audio/destroy,s<mixer1> to remove it, and so on.

[manicksan]

note. this was supposed to be a response before #14
so this is my thoughts

Don't think we need separate categories for "setter"/"getter" functions.
as when we send the OSC message /audioAnalyzeRMS1/available
it should just respond with the return value of the function.

But we need them to separate the

OSC messages to place, rename, remove, connect, disconnect and control audio objects live.

ex (setting value 1.0 to ch0 of mixer1).
/ctrl/mixer1/amplitude/0 1.0

/add/mixer2
/connect/mixer2 out 0 i2s in 0
/connect/mixer1 out 0 mixer2 in 0

I cannot say this is the correct syntax for OSC messages as I don't have read the spec. yet.

I was thinking that it would be possible to use the Web Serial API

It works without secure context when running in offline/local,
also the github pages where I host the tool is in https so that should not be any problem.

I have earlier been looking at the "Web MIDI API" but with no success,

also the downside with "Web Serial API"/"Web MIDI API" is that they are only available in Chrome/Edge/Opera


My workaround is a local running web socket server written in JAVA (currently as Arduino IDE plugin, but can be run standalone as a MIDI bridge)
to make it possible to communicate with MIDI devices even in Firefox

I was planning to add Serial Port support for that server as well.


What are your thoughts about the server, what language should it be written in?
Languages I know:
JAVA (native MIDI/Serial support)
.NET C# (native Serial support)
Node.JS JavaScript/TypeScript (native Serial support)

Languages I don't yet know:
Python 3
GO


by the way about the autogenerate function list
It now exports "function execute by string" code for each of the
AudioStream objects available + complete type def "list".

at the moment it don't support multiparameter functions
and such functions are removed by the generator.
here is a shortcut so that you can view the current progress
https://manicken.github.io/?cmd=expo...teFunctionList

[manicksan]

The current problem with extracting the function list from the Tool-"documentation"
is that it don't include the parameter types.

[JayShoe]

Quick thought about extracting the function list. Do we need to? As long as we define the addresses, can't we just create static functions for each audio object manually? We want to automatically create instances so for example mixer1, mixer2, mixer3. But as long as all mixers are type "AudioMixer4" and we have a matching OSCAudioMixer4 object, then we don't need to dynamically create that list. Right?

The only negative is when other objects are added, we just need to then update our library with the OSCNewObject before it starts working.

Tldr: create static definitions for each object manually (IE AudioMixer4). Only automatically understand each instance for example mixer1 and mixer1.

Ps - the edit function on this forum on mobile is broken and deletes posts.... Android chrome browser.

[h4yn0nnym0u5e]

I had problems running Web Serial API with a file/// destination, but I may not have tried hard enough! I think Paul's philosophy of a local file with the GUI on it is quite nice, doesn't tie to user to having an internet connection. Doesn't matter for initial development, of course.

Note that as far as OSC is concerned the Teensy is the "server" - the GUI editor would be one example of a client, and presumably we'd want to extend that in JavaScript as it's already in that language (and @manicksan, you clearly know what you're doing in it!).

OSC spec is here. Essentially can route function calls with parameters to a leaf on the system tree, so very flexible. Technically only the leaf item is a function, but probably .../mixer1/gain/0,f<0.45> could readily be mapped to mixer1.gain(0,0.45). I'm writing the messages with binary values at the end enclosed in <>, and omitting the NULL padding bytes OSC mandates to put everything on a 32-bit boundary.

I've started a process for scraping the function list from the audio library source code, which deals with the problem of the rather variable documentation quality. Agree @JayShoe, it could be done manually, but it'll be pretty tedious. Manual intervention will be required, as exporting some functions makes little to no sense (I think), e.g. the AudioPlay/RecordQueue classes. If this gains traction then we'd hope people implementing new AudioStream objects would also create its derived AudioOSCStream object.

Off soon to the company Christmas do, so radio silence for a bit - and quite likely a sore head tomorrow...

[manicksan]

I did try download the example from
https://unjavascripter-web-serial-example.glitch.me
and with success run it offline.


the following regex works to extract all function names + parameters
I use it for the AutoComplete functionality

/\s*(unsigned|signed)?\s*(void|int|byte|char|short|long|float|double|bool)\s+(\w+)\s*(\([^)]*\))\s*/g

here is the complete code in javascript

function getFunctions(functionNode, completeItems)
{
    var functions = [...functionNode.comment.matchAll(/\s*(unsigned|signed)?\s*(void|int|byte|char|short|long|float|double|bool)\s+(\w+)\s*(\([^)]*\))\s*/g)];
    for (var fi = 0; fi < functions.length; fi++)
    {                    
        if (functions[fi][1] == undefined) functions[fi][1] = "";
        var returnType = functions[fi][1] + " " + functions[fi][2].trim();
        var name = functions[fi][3].trim();
        var param = functions[fi][4].trim();
        completeItems.push({ name:(name+param), value:(name+param), type:returnType, html: "@ " + functionNode.name + "<br> returns" + returnType, meta: returnType, score:(1000)  });
    }
}

there is this online regex tester
https://regex101.com/

edit.
that regex also matches private functions, so they needs to removed manually, or the code needs some pre-parsing to remove private: sections

[JayShoe]

I had Agree @JayShoe, it could be done manually, but it'll be pretty tedious. Manual intervention will be required, as exporting some functions makes little to no sense (I think), e.g. the AudioPlay/RecordQueue classes. If this gains traction then we'd hope people implementing new AudioStream objects would also create its derived AudioOSCStream object.

Why wouldn't the AudioPlay class make sense to have control for? Sure, it would be useful to have the ability to do simple tasks as well as advanced.

Even the sgtl5000 control would benefit from having micGain, and inputSelect... It's an "I2C control" as opposed to DSP control but useful nonetheless.

I read the spec and I'm also intrigued by the ability for wildcard matching on the OSC address. The spec seems to suggest that one could call /teensy1/mixer*/ch*/ and change the volume level on every mixer in teensy1. It even seems like a string search should be supported too. For example calling simply [mixer] should match all mixers. Calling {mixer, sine} would match both. Very interesting.

[JayShoe]

On that note however... We are discussing mapping the addresses and building the address table. Each object will have different expectations. The mixer expects ch0-3. Sine expects frequency and phase. Are you expecting to autogenerated that too? I would think (and I could be wrong) we need to pay attention to the specifics if each object. So really each object needs to be defined manually, right? Then the documentation must describe what each address will expect (existing audio lib documentation?)

[manicksan]

By autogenerated address we don't really need some special documentation as every object then follow the standard naming scheme. It's also much easier to maintain as all code is generated from one place, and eventually bugs can be fixed at one place instead of going through all objects.

[manicksan]

I was thinking that typically the className would be unnecessary for most uses: if your mixer1 is an instance of the AudioMixer4 class, the OSC address doesn't need the className to refer to it. However, for static functions then it would potentially be useful to have access via the className, which could take the place of the instance name.

I have been thinking about this and when doing dynamic design using OSC
my Audio Design Tool can separate the design into different parts "classes" then all the objects can be put into the same pool and use "virtual classes" to separate OSC addresses, as for example a mixer in classA and classB can have the same name.
In the pool they can have a combined name
classA_mixer1 and classB_mixer1.

I don't know what you mean about

the OSC address doesn't need the className to refer to it

[manicksan]

Also I think we should use the
https://github.com/CNMAT/OSC
as it takes care of the "matching" address to object part

The only downside (or maybe not) with it is that for every message it receives
there need to be a new OSCMessage created
and then the actual matching is by using that message object

OSCMessage msg("/a/1");
msg.dispatch("/a/1", dispatchAddress);

this looks to me very wasteful

as it could be defined in a (OSCaddr -> dispatchAddress) array instead
and when a new message arrives it goes throught that array to find the correct OSCaddr and then the corresponding "dispatchAddress" function can be called.
just like this webserver
https://tttapa.github.io/ESP8266/Cha...%20Server.html

example:

server.on("/", HTTP_GET, handleRoot);     // Call the 'handleRoot' function when a client requests URI "/"
  server.on("/LED", HTTP_POST, handleLED);  // Call the 'handleLED' function when a POST request is made to URI "/LED"

[manicksan]

Ok by reading the source code
I found the empty function, so that means we can reuse the message container,
also there is room for custom functionality
as the functions used by route and dispatch are public.