Do I have a "Thread-Safe" issue? aka: How do Interrupts stop main loop

[Uija]

Hey!

As further tests showed, the issue might be somewhere completely different (on the second MCU), so I will have to check deeper before it makes sense for you to invest time into reading and thinking here!

I am working on a Midi-Sequencer, and have an issue, that the Program freezes, but the Teensy itself seems to operate still. Now I try to find out, why.

What does the Device do?
There are tons of buttons (54) and encoders(19), that are handled by a second Microcontroller (for IO reasons as well as timing reasons for the encoders), that sends Input Data to the Teensy via Serial. The Teensy reads the input, updates data and renders the informations onto a Screen via the ILI9341_T4 Api, als well to RGB LEDs using Pauls WS2812Serial Library.
To have a tight and Main-Loop independent MIDI-Performance, I have a InterruptTimer with short intervals (100micos) running. it calls a method that checks the elapsed micros since the last MIDI-Execution, and if a value is reached, triggers a MIDI-Tick, that takes up to 60micros.

What happens?
Sometimes, when I chance values via the interface while the sequencing is running, the Interface freezes. No Input and/or output is occurring anymore. But the Sequencing happily continues, playing all the notes and modulations like it is intended, so its not the app itself crashing, but somehow the Main Loops seems to be freezed.

This does only occur, when the sequencing is running and I change values, but it is far from happening always. It has not occurred once, when the sequencing was running without me changing values, nor did it occur, when I changed values without the sequencing running.
I tried to "play" around with it (as good as I can do it considering the time every operation takes) and it seems, that, if I change values, the sequencing process does not use, everything is fine, but when I change values the moment, the sequencing process accesses the data, it can occur.

So from my experience with projects outside of the micro-controller-world, this might be a thread-safe-issue. I went into this, with the idea "Microcontrollers are single threaded, what can go wrong", but It seems as it can be a lot

Sadly I don't really understand, how interrupts are interrupting the execution of code etc. Is it interrupting in a way, that the next CPU cycle is already doing interrupt stuff? is it waiting for the current command has finished?

I have a few things in mind, I can test and change, but I am sure there are ways out there, that I don't know/think of?!

1. I will stop the interrupt timer every time I execute the midi process and restart it after it finished, to see, if the interrupt-tick freezes too at that moment
2. I will do some measurements to see if I can integrate the sequencing into the mainloop without having an unreliable midi clock
3. I will see if I can refactor the code in a way, to disable irq every time I change a value (right before I assign a value to the variable and enable it right after it) without having to do that all over the place.

Sorry for that long wall of text. I am thankful for every input I can get. Maybe I am looking at it wrong? Maybe my conclusions are wrong, and its no threading-issue?

Thanks, Jens

 

[defragster]

Not seeing any code doesn't allow any real understanding of what the trouble might be. Assuming this is a T_4.0 or T_4.1?

Is the timer interrupt the only one in use?

One thing to try - given the timer interrupt is each 100 micros: Don't enable the interrupt, but instead test with an elapsedMicros for the elapsed 100 us to called the current _isr() code that does the MidiTick?
> that would eliminate that question. If loop() code is tight then it should be running 100K if not some millions of cycles per second - unless the display code is limiting that?
> if the display code is doing updates that take too long, put a copy of the elapsedMicros testing there to make sure it gets executed

When an interrupt is triggered, the processor may be heading to the _isr() on the next instruction - so any work in progress will be placed on hold.

 

[Uija]

Hey!

I am using a Teensy 4.1. I only use one interrupt time, but I cannot say, if the Display DMA implementation or the LED DMA Implementation or something else uses them too.

I did some more testing: Updating the screen takes up between 2000 and 10000 micros, dependent on how much of the screen changed. As I have to call the midi-tick every 4000 to 9000micro seconds, that will cause trouble...

BUT:

During my measurements (that are printed on the screen, when the app decides that changed data needs the screen to update), I found out, that even when the Freeze happens, the screen is still updated. As the measurements and the screen updating is done in the main loop, the main-loop is still running, so this might be an issue with the other mcu causing trouble and somehow all my testing matched that thread-thing.

So thanks for the answer and peoples time to read it, I will need to make sure the other MCU is running fine first.

 

[KurtE]

Good luck,

In cases like this there are a few general things, that I might look at: (Been bite before!)

a) Make sure variables that are changed in interrupts and used elsewhere are marked volatile.

b) Look for code that is something like this:

if (event_happened) {
    process_the_event
    clear the event
}

Again that is very generic. But suppose you have something like: volatile new_data_arrived = false;

Now suppose:
if (new_data_arrived) {
Process_data();
new_data_arrived = false;
}

And simple interrupt handler:
void my_isr() {
new_data_arrived = true;
}

Works great in most cases. But suppose that in some case, the next interrupt happened while you are processing data,
you will then blindly clear the status and not realize that the new data arrived.
Again, may be random, like only happens if the update of the screen ISR happens during the same time frame as another interrupt, ...

 

[Uija]

Hey!

After playing around, finding a way to reduce render time for the display, putting everything into one loop, and setting the display-update method to "skip frame, if DMA-magic is still running", I still had freezes. Freezes that were display only. Sequencing was playing notes and modulation, I was able to start and stop it as well as press buttons and see reactions on all the RGB LEDs....
So I wanted to contact @vindar, if he knows about an issue with the update function running into a dead end, to find out, that there was indeed an issue with Interrupt Flags not handled correctly. So I updated to the latest version and voila: No freezes anymore.