Interrupts and float operations

[myzb]

Hi,

I'm processing data from 2 IMUs with the Teensy 3.6. I followed the best practices with interrupts and set it up like this:

volatile int flag1, flag2;

void irs1() {
    flag1 = 1;
}

void irs2() {
    flag2 = 1;
}

loop() {

    if (flag1) {
        // read IMU per SPI, save to float data1[10] array    
        flag 1 = 0;
    }

    if (flag2) {
        // read IMU per SPI, save to float data2[10] array    
        flag 2 = 0;
    }

    /* process in 1ms intervals */
    if (micros() - last_timestamp > 1000) {
         // sensor fusion calculations: use data1[10] and data2[10]
         // takes about 10 micros to complete
    }
}

The Problem I'm having is that some of my helper variables inside my sensor fusion functions are getting overwritten. I did some research and found this (see #4 and #5) about floating point values and interrupts:
https://community.arm.com/processor...ng-point-unit-on-the-arm-cortex--m4-processor

From what I understand I may need some extra stack space for when my processing function get interrupted by either IRS??
I tried another approach and moved the sensor-fusion stuff to the "if statement" for flag2 (Like the code below). When I do that, my calculations give me the proper results.

    if (flag2) {
        // read IMU per SPI, save to float data[10] array    
        flag 2 = 0;

    /* process at rate of arrival of interrupt flags */
         
         // sensor fusion calculations: use data1[10] and data2[10]
         // takes about 10 micros to complete
    }

I can just stick to that second approach and be happy, I just want to understand why the other way does cause issues.

EDIT: Found out about something called "float re-entrancy" ... apparently interrupting functions like sin/cos can cause weird behaviour? Still don'understand this fully.

Best regards

 

[defragster]

Since both isr()'s only touch an int - that should not hurt pending FPU operations.

What may be happening is the isr()'s are interrupting the imu data fetching?

Perhaps using noInterrupts(); just before the IMU calls and then interrupts(); just after as a quick test?

#define interrupts() sei()
#define noInterrupts() cli()

Are both IMU's on the same bus? They may be interfering with each other somehow. Given that it works just after flag2 is set at least one of the devices would be 'between int triggers' at that time.

 

[myzb]

I tried using noInterrupts()/Interrupts() already without success.

I just found something called "float re-entrancy" still trying to understand it ... But apparently some functions like sin/cos which I'm using to process my data have some problem when they get interrupted? Anyone knows if that could apply here?

 

[defragster]

By default the FPU registers are not saved { unless required and interrupt code as shown does not use FPU } - because unless another FPU operation is done during the interrupt those registers are not disturbed. If that is not the case they would need to be pushed which would add to interrupt latency { this is detected and done with processor 'Lazy Stacking' enabled in Teensy build }.

If you know the rate of IMU completion you could try it without interrupts and just hit them every x milli/micro seconds between reads and expect the data to be ready without wait. Depends on what IMU youd are using - but that is how onehorse wrote his code - and when I enabled interrupts it worked as well, but that was a single imu.

Something odd is going on - I may not have considered it correctly - but something is afoot - are both imu's on the same bus? SPI or i2c? Other details?

It could be that the instant the int is reported that data is valid but not buffered, so reading it some time later may catch the imu unit mid update when the presented data is not valid. IIRC the onehore code does a query that data is ready. Or perhaps that is where the FIFO comes in on his unit so prior data is set aside from the active in process data.

I would back up and get each IMU working reliably one at a time to make sure you are following the data sheet specs to get valid data on each IMU and then make sure that works when adding the second unit to make sure there are no bus collisions because CS is not respected by one unit or the other being properly addressed/selected - again that depends on if they are SPI ( CS ) or i2c ( unique address ).

 

[myzb]

thanks for you quick reply.

The IMUs are the MPU9250 and are on different Busses. The interrupt signals the moment data is ready to be collected, thus I set the "flag1" and "flag2" variables. In the main loop I read the registers per SPI (blocking) and save it to a local data[10] array.

What do you mean by the FPU registers not saved? Does this mean that if the very simple interrupt that sets the integer flags interrupts the float processing in my main loop I will lose the data? If this is the case it would explain the behaviour.

 

[defragster]

thanks for you quick reply.

The IMUs are the MPU9250 and are on different Busses. The interrupt signals the moment data is ready to be collected, thus I set the "flag1" and "flag2" variables. In the main loop I read the registers per SPI (blocking) and save it to a local data[10] array.

What do you mean by the FPU registers not saved? Does this mean that if the very simple interrupt that sets the integer flags interrupts the float processing in my main loop I will lose the data? If this is the case it would explain the behaviour.

So SPI0 and SPI1 are in use - one imu on each? Separate data and control lines? And software setup to account for that?

If interrupts were disabled across the imu update calls as indicated then ideally that isn't the issue. An interrupt may be missed, but that shouldn't cause corrupted data.

The MPU9250's are mostly what onehorse uses - though on i2c - and his examples on a single unit work well with or without interrupt notification.

The linked note has a good description - the isr() routines shown do not do any FPU operations - those registers should return unaffected and that should not be a factor here.

#4 Using floating point calculations in an Interrupt Service Routine
Floating point calculations are performed on a separate register bank inside the floating point unit. If both the main thread (e.g. main program) and interrupt service routines (ISR) use the FPU, extra context saving and restoring are required to ensure that the ISR does not corrupt the data used by the main thread. The extra context saving and restoring requires extra clock cycles, and therefore if you want to have fast ISR response time, one way is to avoid floating point calculations inside an ISR. In this way, the stacking and unstacking of FPU registers is avoided using a feature called Lazy Stacking (see Arm application note AN298).

 

[PaulStoffregen]

Just to quickly clear a couple things up....

By default the FPU registers are not saved

They are indeed saved.

ARM uses an approach called "lazy stacking" to save the FPU registers on an as-needed basis. Details can be found here.

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dai0298a/DAFGGBJD.html

But apparently some functions like sin/cos which I'm using to process my data have some problem when they get interrupted? Anyone knows if that could apply here?

These C library math functions are supposed to be reentrant. They are very unlikely to be the cause of your memory corruption problem.

If you're working with 32 bit floats, you might wish to use sinf() and cosf() rather than sin() and cos(). The ordinary ones (without the "f") do everything as 64 bit double, which is much slower because it doesn't leverage the FPU.

From what I understand I may need some extra stack space for when my processing function get interrupted by either IRS??

Yes, interrupts allocate 104 or 108 bytes on the stack, if the FPU is active and claims space for lazy stacking. Unless your code is very simple, the compiler will usually push several more registers onto the stack. If your code uses local variables that don't all fit within the ARM registers, they too will be allocated on the function's stack frame. If you call other functions, or do math operations which the compiler implements with library functions (mostly double math), those functions many also grow the stack. So the total stack size used by your interrupt can vary quite a lot, depending mostly on your code. But at a minimum, you can expect about 120 bytes used. Generally the thing to worry about is use of large arrays as local variables, or calling other functions which do that.

If you have interrupts configured at different priority levels, the NVIC interrupt controller will "nest" them. Higher priority interrupts (lower numerical value in the priority register) are allowed to interrupt lower priority ones. The hardware supports up to 16 levels of nesting, but programs using all 16 levels are rare.

By default, interrupts on Teensy 3.x run at priority level 128. The systick interrupt runs at 32, and the USB runs at 112. Hardware serial runs at 64. So 3 or 4 levels of nesting is pretty common.

Even if you have all 16 levels of nesting, and even if all of them use 120 bytes for saved registers and another 512 bytes for local variables, that adds up to a worst case of just over 10K stack memory used, if all 16 happen to interrupt each other! Hopefully 10K isn't a big problem on a chip with 256K of RAM.

Each time you compile, Arduino shows a memory usage summary. The part you want looks like this:

Global variables use 5028 bytes (1%) of dynamic memory, leaving 257116 bytes for local variables.

The 5028 bytes are located at the beginning of the memory. If you use malloc(), chucks of memory are allocated immediately after those 5028 bytes. The stack starts at the end of the 256K and grows downward as you call functions that allocate local variables, and as interrupts execute. If the stack grows too much, it will first begin overwriting anything you've recently allocated with malloc(), and eventually overwrite the global/static variables which are placed at the beginning of the RAM.

Whether that's a problem really depends on your code. But as a general rule, if you avoid large arrays as local variables and don't use malloc(), then the "leaving 257116 bytes for local variables" info Arduino shows can give you a pretty good idea of how safe you are.

 

[PaulStoffregen]

Now, having written that lengthy explanation, I will say we see this sort of question over and over on this forum. The cause almost always turns out to be the same thing: a buffer overflow mistake.

I know it's tempting to question the hardware interrupts, Arduino libraries, and even the standard C library and compiler itself. I'm not saying these are necessarily beyond question. But they are very mature and well tested. They're very unlikely to be the cause of a memory corruption issue.

I'm sure you've looked over your code many times, but I would suggest looking carefully at every array and any place you do pointer arithmetic to write to memory. One very simple thing to try is just making every array unnecessarily larger, until you find the one that "fixes" the problem. Or if you have a huge number of arrays, grow half of them and "binary search" for the one that matters (well, if you have enough extra RAM to make so many larger).

 

[gfvalvo]

This is unrelated to your problem, but I'd change this:

if (flag1) {
  // read IMU per SPI, save to float data1[10] array    
  flag1 = 0;
}

to this:

if (flag1) {
  flag1 = 0;
  // read IMU per SPI, save to float data1[10] array    
}

That way, if another interrupt happens before the processing is finished you won't wipe out the flag.

 

[myzb]

Thanks everyone for the support and feedback. Specially to Paul for clearing up the confusion i had about the FPU stack ... and helping me turning that pointing finger around towards myself. Learned something new.

Yep, as 99% of the time it all boiled to the beloved pointer arithmetic...

I was writing back the normalised sensor data to the same memory where the original data was being stored. The problem was that while most values don't cared about renormalization, one did care...

 

[defragster]

Glad code error was found.

I updated post #4: By default the FPU registers are not saved { unless required and interrupt code as shown does not use FPU } - Lazy stacking was noted in #6 - per OP provided link.