Data types: Which should I be using?

[stereodan]

I'm doing some simple raytracing on the Teensy 4.0. That means doing a lot of dot products of normal vectors, so lots of decimals with values from -1 to 1. I've been using single-precision floats, but I don't think this is the best choice, as I'm seeing a lot of flickering and "stepping" as the results gets multiplied by several orders of magnitude to be displayed on LEDs.

What data types should I be using, and is there a best practice for how the calculations should be handled? I want as much accuracy as possible, as the 4.0 seems to have plenty of speed to handle the workload.

 

[AndyA]

Any reason you don't just switch to using doubles? The CPUs floating point unit includes hardware support for them so it seems like a simple way to get more accuracy for minimal effort or performance hit.

Maybe do

typedef double TraceType;

And then make all your variables / functions use the type TraceType. That way if you want to switch between floats and doubles (or any other data type) you only have to change one line.

You could look at using fixed point maths or an arbitrary precision library. Fixed point will give you the best possible performance for a given resolution while arbitrary precision will as the name implies give you all the precision you want. But they are both far too much effort to consider if switching to doubles does the job.