Arduino Events

PaulStoffregen

Well-known member
For quite some time I've been considering the need for an event callback API in Teensy's core library.

I want to collaborate with Arduino. Eventually I want to contribute this to all Arduino boards, or perhaps all 32 bit boards. But working with Arduino on even fairly simple features often goes very slowly and ends up stalled by many people disagreeing or bikeshedding. My goal is to move quickly and perhaps get a first experimental version into the 1.38 release. So instead of their developer list, I'm going to first compose my thoughts here and ask for your feedback. Contributing back to Arduino can come later...

In designing any API, you can take a top-down approach focusing on the features and functions provided to users, or a bottom-up method focusing on the implementation details. Some amount of bottom-up planning is needed to end up with an efficient implementation, but I'd prefer to start from the top where user needs come first. Over the last year I've talked with several people in person, where usually there's a strong tendency to immediately focus on the difficult low level details. Those things are important, but please for a moment let's think top-down...

Event processing is meant to provide at least 2 major features.

1: Timing sensitive functions can post events to be later processed by other functions, which typically aren't as timing sensitive. In particular, events provide a way for interrupt handlers to schedule non-interrupt functions to later run. Non-interrupt code can generate events too, but the interrupt to non-interrupt case is likely to be the most common usage.

2: A consistent API allows libraries that produce events to be connected to other libraries or user-written code that consumes events. Of course both sides have to agree on the meaning of the events to actually do something useful, but the general idea is some degree of "pluggable" modularity between separately developed code.

This ought to sound pretty similar to GUI programming, where controls or widgets generate events and most programming is done by attaching handler functions. Obviously I don't want to make this as large & complex as Microsoft WIN32. Perhaps events only need to be of a single no-data type? Or maybe some of the features of traditional GUI events would be valuable? I think it's at least worth considering.

Of course this needs to be kept fairly lightweight at runtime, which is where bottom-up design is needed. Posting an event needs to be quick. For example, the Wire library might offer a non-blocking endTransmission() function with posts an event when the transmission actually finishes. Long-term I want to see this event API become the de-facto way asynchronous & non-blocking I/O is done on Arduino libraries.

When events are actually processed involves many trade-offs. The more immediately an event handler function is called, the more disruptive it is to the flow of the rest of your program. Perhaps this is configurable? Maybe when attaching you can specify if you wish to handle events immediately, even at interrupt context. Or maybe there's an option for handling at a low priority interrupt, similar to how the audio library uses 2 interrupt priority levels. Of course, the option to handle events from main program context needs to be present and probably the default. Maybe even there a couple options make sense, like calling from yield() versus calling only after loop() completes? When used together with an RTOS, options for assigning the thread to receive the function call would make sense.

From a style point of view, the code people actually write in sketches wants to have the minimalist simplicity you'd expect from Arduino.

Many low-level implementation details exist. I've tried to keep this top-down. But believe me, I've thought a lot about little issues, like locking to prevent recursive event calls, linked lists between C++ objects to avoid malloc/new dynamic memory, calling static vs C++ class member functions, and so on. This stuff matters, so please by all means comment, but let's try to do so with can-do attitude to solve technical challenges, and with keeping the big picture in mind.
 
Last edited:
This sounds very interesting and useful! Will help as I can.

It will be interesting to see how this progresses.

Obviously this would be a great mechanism for adding Async support to SPI, likewise Wire. Also potentially Serial.. Would be great if this could replace the serialEvent, stuff such that the user is not hit for lots of things (checking events), which there is no consumer for.

Again count me in!
 
Here's roughly how I'm imaging the event API might look...

Code:
EventScheduler myevent;

void setup()
{
  pinMode(13, OUTPUT);
  myevent.attach(dosomething);
}

void loop()
{
  Wire.requestFrom(112, 2, myevent);
  digitalWrite(13, HIGH);
  delay(250);
  digitalWrite(13, LOW);
  delay(250);
}

void dosomething(EventSchedulerRef event)
{
  int num = event.getStatus();  // the final return status of Wire.requestFrom
  if (num == 2) {
    byte b1 = Wire.read();
    byte b2 = Wire.read();
    // do something with the data
    // default attach means we get called
    // from non-interrupt context, so we
    // can use Serial.print, String and
    // non-reentrant Arduino libs.
  }
}
 
Last edited:
Looks useful. Two questions:

- Wondering what happens if you call Wire.requestFrom while you are still in dosomenting()? Will the events be captured or ignored ?

- Would it be possible to add context to the event? I.e. would it be possible to call a member function instead of a plain c function?
 
There needs to support for a void* user context. E.g. something like the Encoder attachInterrupt() usage should be possible, without jumping through hoops. The context can be a synchronization conduit for interrupt-based events. Initially, the event sender has ownership. When the event is delivered, ownership of the context passes to the receiver.

There should also be support for event priorities. Quite a few people have high priority things that need to be processed quickly, while other stuff can wait. The implementation is trivial and low overhead. The C++ standard library has heap functions that work with arrays (the heap would be a priority queue), no dynamic allocation is required.

The event state could be something like:
Code:
struct EventState {
    int status;
    int priority = DEFAULT_PRIORITY; // some people won't care
    callback_t receiver;
    void* context = nullptr;  // some people won't care
};
 
There needs to support for a void* user context. E.g. something like the Encoder attachInterrupt() usage should be possible, without jumping through hoops. The context can be a synchronization conduit for interrupt-based events. Initially, the event sender has ownership. When the event is delivered, ownership of the context passes to the receiver.

There should also be support for event priorities. Quite a few people have high priority things that need to be processed quickly, while other stuff can wait. The implementation is trivial and low overhead. The C++ standard library has heap functions that work with arrays (the heap would be a priority queue), no dynamic allocation is required.

The event state could be something like:
Code:
struct EventState {
    int status;
    int priority = DEFAULT_PRIORITY; // some people won't care
    callback_t receiver;
    void* context = nullptr;  // some people won't care
};

would it not be useful to add an explicit pointer to data to allow the event-handler to call "receiver(context,data);" ?
this would be very useful for data processing, At least, I'm doing it this way.
Or do I misunderstand the purpose of the event concept.
Obviously, data could be part of a context struct.
 
would it not be useful to add an explicit pointer to data to allow the event-handler to call "receiver(context,data);" ?
this would be very useful for data processing, At least, I'm doing it this way.
Or do I misunderstand the purpose of the event concept.
Obviously, data could be part of a context struct.
The 'void* context' in my struct is the user data pointer. It would either be in addition to the EventSchedulerRef thingy or some extra field in it.
 
It looks interesting. I have some of the same questions and will be curious on the details.

In your example, are the events one shot type of things? does the Wire.requestFrom(112, 2, myevent);
simply reset the internals of the myevent to not triggered. and/or work more like counting semaphore?

I personally like the idea of void* as it does give you the option to pass in a this pointer...

When and how are these event handlers triggered? Can some of them be at the time... on Interrupt context?
Or are they handled by yield()? I assume not by something like IntervalTimer as that would be on interrupt context, which may defeat the purpose?

Are you thinking of re-implementing some of the things like serialEvents to use this? If so would it still work like: if (Serial.available()) serialevent();
Or do you see, having the hardware Serial code set the event when new data arrives...

Again looks like it could be fun! And Useful
 
The 'void* context' in my struct is the user data pointer. It would either be in addition to the EventSchedulerRef thingy or some extra field in it.
OK I understand,
so far, for my own scheduler I use the following structure
Code:
typedef   void *Ptr;
typedef   void (*Fxn_t)(Ptr,Ptr) ;

typedef struct  
{      Fxn_t   funct ;        // function to run when the job is scheduled
        Ptr       context ;	     // private information for the job
        Ptr       data ;         // pointer to data to pass to the job when it runs
        int       nice;           // store for nice value (-1 is immediate execution)
} Task_Obj ;
This allows a direct or scheduled callback from, say a dma ISR, with the data pointer pointing explicit into dma vector and the context points to a struct containing relevant info about data.
this structure is good also for my 'PIT' and 'PDB' type activations

For my SWI I extend this struct by other parameters (to be more event-like).
Code:
typedef struct 
{      Task_Obj job;    // job to be executed
        int irq;              // number of interrupt irq 
        int count;          // number of possible executions of job (-1 keep job forever) 
} SWI_OBJ;

This is very close to your suggestion.
I look forward to see where the Arduino-events initiative will end up.
 
I've been contemplating what to call this. In msg #3 I wrote "EventScheduler", but I don't really like that name.

Perhaps the class name is superficial, but I believe the saying "you only get 1 chance to make a first impression". The class name is what most people will see first when looking at example code. It's (probably) what we'll end up calling this feature. A well chosen name could greatly improve the chances Arduino might adopt this API someday.

My hope is a name that says what the object *does*. I want to avoid plain nouns that say what it *is*, especially in technical terms, without giving much idea of what it will do for you (unless you already know what that particular type of thing is and how it would work). When someone does a google search for Arduino code and finds 5 examples that demonstrate how to do concurrent tasks, my hope is only a brief glance at the name gives them a sense this object does something they'll find useful & helpful.

Any ideas or suggestions?
 
All details are still open for discussion, so please don't take any of these answers as inflexible set-in-stone decisions. They're merely what I'm contemplating at this early stage.

- Wondering what happens if you call Wire.requestFrom while you are still in dosomenting()? Will the events be captured or ignored ?

Triggering another event from the called function should be fine. The implementation should complete everything before actually making the call, so you can retrigger the event from within the handler.

- Would it be possible to add context to the event?
There needs to support for a void* user context
would it not be useful to add an explicit pointer to data to allow the event-handler to call "receiver(context,data);" ?

Yes, agreed, an optional context pointer is essential. It'll probably be accessed with getData() and setData() members, or maybe just expose the actual pointer? Some people may use it to pass per-event data, but the intended use will probably be static context for whatever the event is about. Or maybe 2 such context pointers would be needed?

I.e. would it be possible to call a member function instead of a plain c function?

Yes, I believe calling a non-static member function should be supported. Anyone have any suggestions on C++ syntax to do so?

There should also be support for event priorities. Quite a few people have high priority things that need to be processed quickly, while other stuff can wait.

I've been thinking along the lines of 4 or 5 options for ways the function is called. Immediate would just call the function. Others would probably implement in-order queues. A software interrupt (perhaps ARM PendSV) would allow higher priority than main program, but lower than most other interrupts. Two or maybe 3 options for yield() might be possible, maybe one that always calls from any yield, and another only if the main program is in delay(), and maybe even a lowest priority like only when loop() returns?

Do you believe assigning an integer priority and re-ordering the list of pending events is really needed? I'm concerned that's adding quite a bit of complexity, even if there's a nifty C++ syntax.

In your example, are the events one shot type of things? does the Wire.requestFrom(112, 2, myevent);
simply reset the internals of the myevent to not triggered. and/or work more like counting semaphore?

I was imagining the simplest case, just one-shot. If you trigger the same event 2 or more times before the callback, the function gets called only once.

OK I understand,
so far, for my own scheduler I use the following structure......

One big question in my mind is whether the extra overhead of virtual functions is worthwhile, to allow anyone to extend it and add their own scheduling methods in their inherited class?
 
Keywords above are: callback, function, process, request, generate

It is less 'scheduled' than requested: EventRequest
 
Name: Not sure - EventScheduler - does not feel right in that I don't think it is the object that is actually doing the scheduling, but more just something like:
Event, or EventState, or EventRequest...

But may depend on how you see the event producer working... That is with: Wire.requestFrom(112, 2, myevent);
Does the requestFrom hold own to a pointer/reference to myevent and then call a method on it to trigger it? Or is there a top dog Event scheduler object that myevent is registered with and which the producer of the event talks to?

Also wonder if you see usage pattern, where the user could choose not to set callback function, but instead, have the ability to query the state?

Code:
  Wire.requestFrom(112, 2, myevent);
... 
  if (myevent.isTriggered()) {
 ...
}

Likwewise maybe some form of wait, or the user can do them self... Example assume converted SPI async code to use events. Now I wish to do a simple update screen for Teensyview, which requires a 6 byte header output, followed by 512(or 1024) byte output of the data... Now in my program, I may wish to start the screen update(async), than allow the program to go do some different stuff and then when that finishes update screen again. So how should the user code know the display update has completed and/or wait for it to complete?
a) All internal to display - Have methods in the library, like: isAsyncUpdateActive() or waitAsyncUpdateToComplete()...
b) user call Update with Event object and the have callback function
c) like b) but can query/wait...

If the class has some form of wait, does this extend to be able to wait on multiple events, maybe with timeouts (linux select )

Do you see a way to use this to transparently replace some of the current stuff like serialEvent?

Or are there some form of SystemEvents that maybe someone can register to by number/mask? I am guessing probably not (KISS). Example if HardwareSerial for Serial1, receives a character on ISR, should it check for some registered Event object and trigger it? Or can it potentially Simply set/trigger something regardless, or should it stay like today where yield does calls to Serial1.available()...

Oops sorry - Maybe a little too many thoughts here...
 
an event scheduler would need a backend buffer of some sort, to queue the data to parse later in the event handler
the only issue with this is if you have an mcu lacking memory space..., that and, if the queue isn't emptied fast enough, it'll either overflow (unless coded properly), or just stop queuing additional data till the buffer clears a bit (thereby loosing new queue data..)

I've re-written a library that used this type of approach, but the event function of the library always has to be called in the loop() function while the handler code waited for the events, but that alone uses (pre-set) a 16 frames of 6 byte queue for each event
CMD, OBJECT, INDEX, (<uint , 16_t ), CRC <-- frame of 6 bytes
 
Last edited:
I just re-read the original API proposal in #3. Does this mean that you need an EventScheduler for each and every event you want to subscribe?

I'd prefer something like shown in the code block below. Classes which want to provide events simply add as much Event members to their public interface as they need. If they want to fire an event they call Event::fire(), class users who want to subscribe call Event::subscribe(handler_t) Event::Unsubscribe or similar functions.

Code:
class myClass
{
public:
    Event  dataAvailable;  // this event will be fired if we have new data 
    Event  errorDetected; // another event for fired in case of an error
    //... and so on

    void doSomething()
    {
        // ...
        // something happened
        // errorDetected.Fire();  // this would call all subscribed handlers of this event
    }
};


myClass mc;


// define some event handlers (omitted context for simplicity) Would be perfect if non static member functions could be used also.

void gotData()
{
    // handle data
}

void panic()
{
    while (1) blink();
}

setup()
{
    // In case I want to listen to an event I simpliy subscribe to it
    mc.dataAvailable.subscribe(gotData);
    mc.errorDetected.subscribe(panic);

   //work with myClass....
    
    // don't need the event anymore...
    mc.dataAvailable.unsubscribe(gotData);

}
 
Here is a first attempt, so far calling it "EventResponder".

https://github.com/PaulStoffregen/cores/blob/master/teensy3/EventResponder.h
https://github.com/PaulStoffregen/cores/blob/master/teensy3/EventResponder.cpp

Keep in mind the main purpose is to give you way to control how your handler function gets called, *without* editing library code that triggers the event.

The default attach method causes your function to be called from yield(). This is the main usage case, where the event is triggered from within an interrupt. Calling from yield lets you access ordinary data without volatile and blocking interrupts or other tricky synchronism. You can safely use almost any Arduino function and library, though blocking or delaying will prevent other events from being serviced until your handler finishes.

If you want your function called immediately even if from interrupt context, you can specify that. The idea is this decision about how your function gets call does not get coded into libraries like Wire and MillisTimer (which I'm going to do soon...) You choose at runtime which way you want it done.

Another option to have your function run from a software interrupt isn't working yet. I'll finish it soon. This lets you have your function do work at an interrupt which runs above main program context, but at a priority lower than most other interrupts so you don't impose latency on timing critical hardware response. It also gives a way for main program code to trigger something to run at software interrupt level. Or it will... soon.

To answer a few of the above questions:

Does this mean that you need an EventScheduler for each and every event you want to subscribe?

Yes, it does. But you can attach the same function to many of them, so they are all handled by the same code.

an event scheduler would need a backend buffer of some sort, to queue the data to parse later in the event handler

So far, even event carries 2 pieces of data, an integer status code and a pointer to data. So if you want to pass data, you can. But this layer is very thin... it's up to you to craft a scheme where the event creator allocates the data and the handler consumes and releases it.

You might notice I made a couple of the functions virtual. I must admit at this very early stage it's merely an unexplored idea, but the intention is to allow you to derive your own class which lets you do more. Whether that's even feasible, I'm not sure. Looking for feedback....

If the class has some form of wait, does this extend to be able to wait on multiple events, maybe with timeouts (linux select )

I put in placeholder declarations for wait functions. So far, not implemented.

Do you see a way to use this to transparently replace some of the current stuff like serialEvent?

Yes, I absolutely intend to replace serialEvent, even though it's not done yet. Very likely it will be still a weak function for compatibility, but the event attach will happen in SerialX.begin(). No more overhead for checking 6 ports on every yield(), even if none are actually used.

There needs to support for a void* user context.

I put this in. Admittedly it's just very simple setContext() and getContext(). It's separate from a data pointer, which is meant to be per-event data.

There should also be support for event priorities. Quite a few people have high priority things that need to be processed quickly, while other stuff can wait.

The idea is 4 different event types, called immediately, called from a low priority software interrupt, called from yield(), or polled.

I wanted to keep this simple. So far there isn't any provision to call the functions within one level in any order other than the order their events were triggered. I'm a bit skeptical re-ordering the lists of waiting events would be worth the extra overhead. But now's the time to consider it...

A 5th placeholder for RTOS system is also in the code, intended for running an event function as its own thread. I'm looking for feedback from people using RTOS systems about how this should be done in practice.
 
Sounds great!

I will take a look through this and see how hard it would be to convert my version of the SPI library that has Async support over to using this instead of callback functions.

Then the question will be, do we (I) need to do another round of email on the Developers List?

Edit: For example currently in my SPI library I have the two Async functions:
Code:
    bool transfer(const void *txBuffer, void *rxBuffer, size_t count, void(*callback)(void));
    bool transfer16(const uint16_t * buf, uint16_t * retbuf, size_t count, void(*callback)(void));

Do you see these changing to be more like:
Code:
    bool transfer(const void *txBuffer, void *rxBuffer, size_t count, EventResponder *event_responder, void* event_data);
    bool transfer16(const uint16_t * buf, uint16_t * retbuf, size_t count,  EventResponder *event_responder, void* event_data);
I used pointer as I have found that not all Async operations need callbacks, although they typically want a way to find out it completed.
 
Last edited:
Then the question will be, do we (I) need to do another round of email on the Developers List?

Not just yet, but soon, yes. I'll make the post. Believe me, I'm not looking forward to the inevitable bikeshedding and design-by-committee talk, but the long-term goal is to contribute this to Arduino.

However, we're not going to wait. This absolutely is going forward on Teensy. The only point of more discussion on their mail list is to see if they feel any changes are essential. If so, we'll very likely go with it. But if they just never decide anything, we're not waiting.

I'd image the SPI async API would probably look like this.

Code:
    bool transfer(const void *txBuffer, void *rxBuffer, size_t count, EventResponderRef event_responder);

Internally you'd store the EventResponderRef as a pointer, but pointer syntax is generally not accepted for Arduino APIs so a reference is used to pass it in. Then later when the operation completes, call its triggerEvent() function.

I don't see any need to pass another pointer in this case, since there's already 2 for the data. The event supports a data pointer, but most cases probably won't make any use of it.
 
Thanks Paul, I started playing with it, currently have the extra pointer in it.

Have it compiling, then updating my Teensyview (ssd1306) code to make use of the updates, and then found as you mentioned that I don't need the extra data, as I was going to pass this pointer, but that is already served by having the main object setup of the EventResponder set the this in the context pointer (setContext).

So I will remove it!

Thanks Again,

As for email list...
 
So I'm confused, what happens when I call "yield" from a function that is called from "runFromYield"? Does that "yield" get ignored and the current function runs until completion?
 
what happens when I call "yield" from a function that is called from "runFromYield"?

It does nothing, because of this "runningFromYield" check:

Code:
        static void runFromYield() {
                EventResponder *first = firstYield;
                if (first && [B]!runningFromYield[/B]) {
                        runningFromYield = true;
                        firstYield = first->_next;
                        if (firstYield) firstYield->_prev = nullptr;
                        first->_pending = false;
                        (*(first->_function))(*first);
                        runningFromYield = false;
                }
        }

Does that "yield" get ignored and the current function runs until completion?

Yup, this code prevents recursive event handler calls. While your event handler is running, you can use functions that call yield(). Of course, doing this will (probably) wreck the response time to other events, but that's your choice. You shouldn't call stuff like delay() inside an event handler, but you can if you like.
 
This is different from what I expected.

Since there already is the EventResponder instance for each event (which is passed to the callback function), the context pointer is redundant. People could just subclass EventResponder if they want additional data inside.

I wanted to keep this simple. So far there isn't any provision to call the functions within one level in any order other than the order their events were triggered. I'm a bit skeptical re-ordering the lists of waiting events would be worth the extra overhead. But now's the time to consider it...

With the intrusive lists, easy/efficient reordering would be an issue. How about having 2 event chains for yield, e.g. EventTypeYieldLowPriority, EventTypeYieldHighPriority? While EventTypeYieldHighPriority is not empty, only stuff from that list is run.
 
Back
Top