float <-> unit32 punning

[DavidBell]

I am sending some floats formatted as hex strings over serial between a couple of teensy boards, and there is a PC on the same comms line.

I do this as follows (Teensy 3.6 over Serial1 @ 500Kbps)

// Sender

    uint_32_t id = 1;
    float val1 = 12.37;
    float val2 = -19.25;

    printf("%08lX:%08lX,%08lX", *(uint32_t*)&val1, *(uint32_t*)&val2); // use uint32 punning

    // Send: 00000001:3E5D0FA2,BEABFFB0

// Receiver

    // Receive: 00000001:3E5D0FA2,BEABFFB0

    uint32_t uid;
    uint32_t uval1, uval2;
    float fval1, fval2

    scanf("%08lX:%08lX,%08lX", &uid, &uval1, &uval2);
    fval1 = *(float*)&uval1;   // pun it back to float
    fval2 = *(float*)&uval2;

    printf("%04u: %03.2f, %03.2f", uid, fval1, fval2);

    // Result on PC:  0001: 12.37, -19.25
    // Result on T36: 0001: 12.37, nan

    // Edit to add. It seems to be negative numbers. +19.25 is fine

Any idea why the negative float value comes back as nan? And only on T36, not PC?

Comms are definitely OK, the correct string is being received at both PC & T36.

I was wondering if T36 float byte format was different to the PC, but then you would expect the PC to show the error.

Thanks in advance.

 

[defragster]

That was a snippet - and had at least 3 typo/compile errors

When fixed and tested as follows it looks to work? Done on a T_4.1 - and the HEX values differ from those shown.

C:\T_Drive\tCode\FORUM\EEPROMread\EEPROMread.ino Jul 12 2022 00:09:24
00000001:4145EB85,C19A0000

0001: 12.37, -19.25

Did a quick search and skipping the Serial TX<>RX with szFoo - after reading this: c-faq.com/stdio/scanfprobs.html
Assume this should work the same on T_3.6?

#include <arduino.h>

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  while (!Serial && millis() < 4000 );
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);


  // Sender

    uint32_t id = 1;
    float val1 = 12.37;
    float val2 = -19.25;
    char szFoo[256];

    [B][U]sprintf(szFoo,[/U][/B] "%08lX:%08lX,%08lX", id, *(uint32_t*)&val1, *(uint32_t*)&val2); // use uint32 punning
    Serial.print( szFoo );

    // Send: 00000001:3E5D0FA2,BEABFFB0

// Receiver

    // Receive: 00000001:3E5D0FA2,BEABFFB0

    uint32_t uid;
    uint32_t uval1, uval2;
    float fval1, fval2;

    [B][U]sscanf(szFoo,[/U][/B] "%08lX:%08lX,%08lX", &uid, &uval1, &uval2);
    fval1 = *(float*)&uval1;   // pun it back to float
    fval2 = *(float*)&uval2;

    Serial.printf("\n\n%04u: %03.2f, %03.2f", uid, fval1, fval2);

    // Result on PC:  0001: 12.37, -19.25
    // Result on T36: 0001: 12.37, nan

    // Edit to add. It seems to be negative numbers. +19.25 is fine

}

 

[PaulStoffregen]

Tried a couple quick tests. I can't get "nan" for 0xBEABFFB0. It always ends up as -0.34, not -19.25.

void setup() {
  Serial.begin(9600);
  while (!Serial) ; // wait
  const char *in = "00000001:3E5D0FA2,BEABFFB0";
  uint32_t uid;
  uint32_t uval1, uval2;
  float fval1, fval2;
  int n = sscanf(in, "%08X:%08X,%08X", &uid, &uval1, &uval2);
  Serial.printf("n = %d\n", n);
  fval1 = *(float*)&uval1;   // pun it back to float
  fval2 = *(float*)&uval2;
  Serial.printf("%04u: %03.2f, %03.2f", uid, fval1, fval2);
}

void loop() {
}

Same result using union rather than casting pointers.

void setup() {
  Serial.begin(9600);
  while (!Serial) ; // wait
  union { uint32_t u; float f; } u;
  u.u = 0xBEABFFB0;
  Serial.printf("u = %08lX, f = %03.2f\n", u.u, u.f);
  u.f = -19.25;
  Serial.printf("u = %08lX, f = %03.2f\n", u.u, u.f);
}

void loop() {
}

Not sure why it's not working for you. I recommend getting your string into a buffer and use sscanf(). Using scanf() probably fails on Teensy because there isn't a concept of "stdio". Likewise for plain printf(), use Serial.printf() as in these examples, or use sprintf() to write to a buffer and then use Serial.write() or Serial.print() to transmit it.

 

[DavidBell]

Sorry about the typos, it was typed out from larger bit of code.

It actually uses vsprintf & vsscanf like this

            int Format(const char* fmt, ...)
                {// wraps sprintf and returns normal character count
                    int res = 0;
                    va_list args;
                    va_start(args, fmt);
                    res += vsprintf(data, fmt, args);
                    va_end(args);
                    return res;
                }

Defragster's code works on a different board I have here, including different hex values.

It's odd that the different values are working on positive numbers.

I'll have to wait till tomorrow to try the troublesome board.

PS Paul. Yes I tried the union thing with the same result.

 

[PaulStoffregen]

If it still isn't working on Teensy, please give us a complete program we can copy into Arduino and run on Teensy 3.6 to reproduce the problem.

Code fragments, especially ones with functions that can't even work on Teensy, tend to lead to a lot guesswork effort. That effort could be much more helpful with a complete program that reproduces the problem.

 

[DavidBell]

Will do. I will be playing with it tomorrow, hopefully will post a snippet or solution.

 

[Nominal Animal]

When punned to 32-bit unsigned integers with little-endian byte order:

12.37:
  Hex: 0x4145eb85
  Dec: 1095101317
  Bin: 0b01000001010001011110101110000101
  Flt: 12.3699998

-19.25:
  Hex: 0xc19a0000
  Dec: 3248095232
  Bin: 0b11000001100110100000000000000000
  Flt: -19.25

0.21587994:
  Hex: 0x3e5d0fa2
  Dec: 1046286242
  Bin: 0b00111110010111010000111110100010
  Flt: 0.21587994

-0.33593511:
  Hex: 0xbeabffb0
  Dec: 3198943152
  Bin: 0b10111110101010111111111110110000
  Flt: -0.33593511

Thus, the error seems to be the hex values you send, 0x3E5D0FA2 and 0xBEABFFB0, that really represent 0.21587994 and -0.33593511, respectively. If you look at the hexadecimal and binary versions, you'll see that it isn't just a byte order issue; besides, you can use Compiler Explorer to verify on ARM for yourself.

Type punning via a pointer is actually undefined behaviour in C. What you should do, is use an union instead, as that's what footnotes in the standard itself says. The following implements u32_to_float() and float_to_u32() for type punning, plus u32_to_hex() for conversion to hexadecimal string (use 8 for length) and hex_to_u32() for parsing a hexadecimal string.

typedef union {
    uint32_t  u;
    float     f;
} word32;

static inline uint32_t  float_to_u32(float value) { return ((word32){ .f = value }).u; }

static inline float  u32_to_float(uint32_t  value) { return ((word32){ .u = value }).f; }

static const char  hex_digit[16] = "0123456789ABCDEF";

static void  u32_to_hex(char *to, size_t len, uint32_t value)
{
    to[len] = '\0';
    while (len-->0) {
        to[len] = hex_digit[value & 15];
        value >>= 4;
    }
}

const char *hex_to_u32(const char *src, uint32_t *to)
{
    const char *const end = src + 8;
    uint32_t  val = 0;

    while (src < end) {
        if (*src >= '0' && *src <= '9') {
            val = (val << 4) | (*(src++) - '0');
        } else
        if (*src >= 'A' && *src <= 'F') {
            val = (val << 4) | (*(src++) - 'A' + 10);
        } else
        if (*src >= 'A' && *src <= 'F') {
            val = (val << 4) | (*(src++) - 'A' + 10);
        } else
            break;
    }

    if (to)
        *to = val;

    return src;
}

You can use the above in both Arduino and hosted C or C++.

To examine such values, here is a small C program that I use to explore various bit and byte order issues:

// SPDX-License-Identifier: CC0-1.0
#include <stdlib.h>
#include <inttypes.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

/* Note: mask = 24 yields byteswap(), reversing byte order. */
static inline uint32_t  bitswap(uint32_t value, uint32_t mask)
{
    if (mask & 1)
        value = ((value & 0x55555555) << 1) | ((value >> 1) & 0x55555555);
    if (mask & 2)
        value = ((value & 0x33333333) << 2) | ((value >> 2) & 0x33333333);
    if (mask & 4)
        value = ((value & 0x0F0F0F0F) << 4) | ((value >> 4) & 0x0F0F0F0F);
    if (mask & 8)
        value = ((value & 0x00FF00FF) << 8) | ((value >> 8) & 0x00FF00FF);
    if (mask & 16)
        value = ((value & 0x0000FFFF) << 16) | ((value >> 16) & 0x0000FFFF);
    return value;
}

typedef union {
    uint32_t  u;
    float      f;
} word32;

static int parse_float(const char *src, float *to)
{
    float       val;
    const char *end;

    if (!src || !*src)
        return -1;

    end = src;
    errno = 0;
    val = strtof(src, (char **)&end);
    if (errno)
        return -1;
    if (end == src)
        return -1;

    while (*end == '\t' || *end == '\n' || *end == '\v' ||
           *end == '\f' || *end == '\r' || *end == ' ')
        end++;
    if (*end)
        return -1;

    if (to)
        *to = val;

    return 0;
}

static int parse_u32(const char *src, uint32_t *to)
{
    unsigned long  val;
    const char    *end;

    if (!src || !*src)
        return -1;

    end = src;
    errno = 0;
    val = strtoul(src, (char **)&end, 0);
    if (errno)
        return -1;
    if (end == src)
        return -1;
    if ((unsigned long)((uint32_t)(val)) != val)
        return -1;

    while (*end == '\t' || *end == '\n' || *end == '\v' ||
           *end == '\f' || *end == '\r' || *end == ' ')
        end++;
    if (*end)
        return -1;

    if (to)
        *to = (uint32_t)(val);

    return 0;
}

static const char *u32_bits(uint32_t value)
{
    static char  buffer[40];
    char        *ptr = buffer + sizeof buffer;
    int          len = 32;

    *(--ptr) = '\0';
    while (len-->0) {
        *(--ptr) = '0' + (value & 1);
        value >>= 1;
    }

    *(--ptr) = 'b';
    *(--ptr) = '0';

    return ptr;
}

static const char *float_to_string(float value)
{
    static char  buffer[1024];
    int  len = snprintf(buffer, sizeof buffer, "%.24f", value);

    if (len < 1 || len >= (int)sizeof buffer)
        return "(Err)";

    while (len > 2 && buffer[len - 1] != '.' && buffer[len - 2] != '.') {
        float  tmp;
        char   old = buffer[len - 1];

        buffer[len - 1] = '\0';
        if (parse_float(buffer, &tmp) || tmp != value) {
            buffer[len - 1] = old;
            return buffer;
        }

        --len;
    }

    return buffer;
}

int main(int argc, char *argv[])
{
    uint32_t  mask;
    word32    src, dst;

    if (argc < 3 || !strcmp(argv[1], "-h") || !strcmp(argv[1], "--help")) {
        const char *arg0 = (argc > 0 && argv && argv[0] && argv[0][0]) ? argv[0] : "(this)";
        fprintf(stderr, "\n");
        fprintf(stderr, "Usage: %s [ -h | --help ]\n", arg0);
        fprintf(stderr, "       %s TRANSFORM NUMBER [ NUMBER... ]\n", arg0);
        fprintf(stderr, "\n");
        fprintf(stderr, "This program converts between floats and 32-bit unsigned ints,\n");
        fprintf(stderr, "applying TRANSFORM to the binary representation.\n");
        fprintf(stderr, "\n");
        fprintf(stderr, "TRANSFORM is a 5-bit number (0 through 31, inclusive),\n");
        fprintf(stderr, "with 24 referring to byte swap, and 31 full bit order reversal;\n");
        fprintf(stderr, "zero is no transform.\n");
        fprintf(stderr, "(First bit in TRANSFORM represents bit pairs, second pairs of\n");
        fprintf(stderr, " bit pairs, third quads of bits, fourth bytes, and fifth pairs\n");
        fprintf(stderr, " of bytes to be swapped.)\n");
        fprintf(stderr, "\n");
        return EXIT_SUCCESS;
    }

    if (parse_u32(argv[1], &mask) || mask >= 32) {
        fprintf(stderr, "%s: Invalid transform (0 .. 31, inclusive).\n", argv[1]);
        return EXIT_FAILURE;
    }

    for (int arg = 2; arg < argc; arg++) {
        if (!parse_u32(argv[arg], &src.u) ||
            !parse_float(argv[arg], &src.f)) {

            printf("%s:\n", argv[arg]);
            printf("  Hex: 0x%08x\n", src.u);
            printf("  Dec: %u\n", src.u);
            printf("  Bin: %s\n", u32_bits(src.u));
            printf("  Flt: %s\n", float_to_string(src.f));
            if (mask) {
                dst.u = bitswap(src.u, mask);
                printf("  Hex[%u]: 0x%08x\n", (unsigned int)mask, dst.u);
                printf("  Dec[%u]: %u\n", (unsigned int)mask, dst.u);
                printf("  Bin[%u]: %s\n", (unsigned int)mask, u32_bits(dst.u));
                printf("  Flt[%u]: %s\n", (unsigned int)mask, float_to_string(dst.f));
            }

        } else {
            fprintf(stderr, "%s: Not a float nor a 32-bit integer.\n", argv[arg]);
            return EXIT_FAILURE;
        }
    }

    return EXIT_SUCCESS;
}

It takes a transformation value (0 = no transform, 24 = byte order swap, 31 = bit order swap), and one or more unsigned 32-bit integers or floats, and describes them as shown first in this message.

None of the bit order transforms yield the hex values for 12.37 and -19.25 that you showed. It would not be the first time GCC generates buggy code when using pointer dereferencing to type-pun values, and I don't see anything else that could be the source of the problem.

 

[defragster]

See post #2: the provided code was buggy and perhaps not trust worthy for the provided values in comments!

Provided code functioned as written ( with punning warnings ) but provided these values in the process:

C:\T_Drive\tCode\FORUM\EEPROMread\EEPROMread.ino Jul 12 2022 00:09:24
00000001:4145EB85,C19A0000

0001: 12.37, -19.25

from:

    sprintf(szFoo, "%08lX:%08lX,%08lX", id, *(uint32_t*)&val1, *(uint32_t*)&val2); // use uint32 punning
    Serial.print( szFoo );
...
    sscanf(szFoo, "%08lX:%08lX,%08lX", &uid, &uval1, &uval2);
    fval1 = *(float*)&uval1;   // pun it back to float
    fval2 = *(float*)&uval2;
    Serial.printf("\n\n%04u: %03.2f, %03.2f", uid, fval1, fval2);

 

[Nominal Animal]

See post #2: the provided code was buggy and perhaps not trust worthy for the provided values in comments!

Right. I have seen Heisenbugs related to type-punning via pointers, but their root cause could have been in (invalid types of) pointer aliasing.

My main point was that in the given code, it was the only possible source. But you are obviously absolutely right: in the bigger picture, the problem is likely somewhere else.

I do use

typedef union {
    uint64_t  u64;
    uint32_t  u32[2];
    uint16_t  u16[4];
    uint8_t   u8[8];
    int64_t   i64;
    int32_t   i32[2];
    int16_t   i16[4];
    int8_t    i8[8];
    float     f[2];
    double    d;
} word64;

typedef union {
    uint32_t  u32;
    uint16_t  u16[2];
    uint8_t   u8[4];
    int32_t   i32;
    int16_t   i16[2];
    int8_t    i8[4];
    float     f;
} word32;

quite often in standard C and POSIX C as well as freestanding C and C++. The standard says these integer types must be exact-size, and the signed versions must use two's complement; and all the architectures I use (almost all currently manufactured, except for some DSPs) use IEEE 754 Binary32 for 'float' and Binary64 for 'double', respectively, so it is a rather practically portable way to convert between floating-point and integer types.

When the data transfers are asymmetric –– say, from a MCU to a computer; or from a computing node to long-term storage ––, it is useful to use prototype values to ensure the floating-point and integer byte order (which may not always have the same byte order, either!) are well defined. In practice, you save in the header, or pass as a Hello! message, one 64-bit signed integer with a negative value, one Binary64 (double), and one Binary32 (float), all with distinct bytes in their values. Then, you can use the byteorder32() function below to check which mask (0 through 3) yields the correct order for the 32-bit, or byteorder64()[/URL] (mask 0 through 7) for 64-bit; and if one does, use that mask for all values you read.

The below program is what I use to explore suitable floating-point constants (in form d/n):

// SPDX-License-Identifier: CC0-1.0
// Author: Nominal Animal
#include <stdlib.h>
#include <inttypes.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <math.h>

/* This program assumes the following:
    - 'float'  type implements IEEE 754-2008 Binary32
    - 'double' type implements IEEE 754-2008 Binary64
    - integer and floating-point types have the same byte order
   It does verify these assumptions in main(), too.
*/

typedef union {
    uint64_t    u64;
    uint32_t    u32[2];
    uint16_t    u16[4];
    uint8_t     u8[8];
    int64_t     i64;
    int32_t     i32[2];
    int16_t     i16[4];
    int8_t      i8[8];
    float       f[2];
    double      d;
} word64;

typedef union {
    uint32_t    u32;
    uint16_t    u16[2];
    uint8_t     u8[4];
    int32_t     i32;
    int16_t     i16[2];
    int8_t      i8[4];
    float       f;
} word32;

static inline word32  byteorder32(word32 value, int mask)
{
    if (mask & 1)
        value.u32 = ((value.u32 >>  8) & UINT32_C(0x00FF00FF)) | ((value.u32 & UINT32_C(0x00FF00FF)) <<  8);
    if (mask & 2)
        value.u32 = ((value.u32 >> 16) & UINT32_C(0x0000FFFF)) | ((value.u32 & UINT32_C(0x0000FFFF)) << 16);
    return value;
}

static inline word64  byteorder64(word64 value, int mask)
{
    if (mask & 1)
        value.u64 = ((value.u64 >>  8) & UINT64_C(0x00FF00FF00FF00FF))
                  | ((value.u64 & UINT64_C(0x00FF00FF00FF00FF)) <<  8);
    if (mask & 2)
        value.u64 = ((value.u64 >> 16) & UINT64_C(0x0000FFFF0000FFFF))
                  | ((value.u64 & UINT64_C(0x0000FFFF0000FFFF)) << 16);
    if (mask & 4)
        value.u64 = ((value.u64 >> 32) & UINT64_C(0x00000000FFFFFFFF))
                  | ((value.u64 & UINT64_C(0x00000000FFFFFFFF)) << 32);
    return value;
}

static inline const char *skip_whitespace(const char *src)
{
    if (src)
        while (*src == '\t' || *src == '\n' || *src == '\v' ||
               *src == '\f' || *src == '\r' || *src == ' ')
            src++;
    return src;
}

static const char *float_bits(float value)
{
    static char  buf[64];
    char        *p = buf + sizeof buf;
    uint32_t     u = ((word32){ .f = value }).u32;
    int          n;

    *(--p) = '\0';

    n = 23;      /* Binary32 has 23 bits in the mantissa. */
    while (n-->0) {
        *(--p) = '0' + (u & 1);
        u >>= 1;
    }

    *(--p) = ')';
    *(--p) = '0' + !!(u); /* Zero if high bits are zero, 1 otherwise */
    *(--p) = '(';
    *(--p) = ' ';

    n = 8;      /* Binary32 has 8 bits in the exponent. */
    while (n-->0) {
        *(--p) = '0' + (u & 1);
        u >>= 1;
    }

    *(--p) = ' ';

    /* Sign bit. */
    *(--p) = '0' + (u & 1);

    return p;
}

static const char *double_bits(double value)
{
    static char  buf[80];
    char        *p = buf + sizeof buf;
    uint64_t     u = ((word64){ .d = value }).u64;
    int          n;

    *(--p) = '\0';

    n = 52;      /* Binary64 has 52 bits in the mantissa. */
    while (n-->0) {
        *(--p) = '0' + (u & 1);
        u >>= 1;
    }

    *(--p) = ')';
    *(--p) = '0' + !!(u); /* Zero if high bits are zero, 1 otherwise */
    *(--p) = '(';
    *(--p) = ' ';

    n = 11;     /* Binary64 has 11 bits in the exponent. */
    while (n-->0) {
        *(--p) = '0' + (u & 1);
        u >>= 1;
    }

    *(--p) = ' ';

    /* Sign bit. */
    *(--p) = '0' + (u & 1);

    return p;
}

static const char *u32_bits(uint32_t u)
{
    static char  buf[64];
    char        *p = buf + sizeof buf;
    int          n;

    *(--p) = '\0';
    n = 32;
    while (n-->0) {
        *(--p) = '0' + (u & 1);
        u >>= 1;
        if (n && !(n & 7))
            *(--p) = '\'';
    }

    return p;
}

static const char *u64_bits(uint64_t u)
{
    static char  buf[80];
    char        *p = buf + sizeof buf;
    int          n;

    *(--p) = '\0';
    n = 64;
    while (n-->0) {
        *(--p) = '0' + (u & 1);
        u >>= 1;
        if (n && !(n & 7))
            *(--p) = '\'';
    }

    return p;
}

int parse_intmax(const char *src, intmax_t *to)
{
    const char *end;
    intmax_t    val;

    end = src;
    errno = 0;
    val = strtoimax(src, (char **)&end, 0);
    if (errno)
        return -1;
    if (end == src || *end)
        return -1;

    if (to)
        *to = val;

    return 0;
}

int parse_uintmax(const char *src, uintmax_t *to)
{
    const char *end;
    uintmax_t   val;

    end = src;
    errno = 0;
    val = strtoumax(src, (char **)&end, 0);
    if (errno)
        return -1;
    if (end == src || *end)
        return -1;

    if (to)
        *to = val;

    return 0;
}

static const char *next_float(const char *src, float *to)
{
    const char *end;
    float       val;

    end = src;
    errno = 0;
    val = strtof(src, (char **)&end);
    if (errno)
        return NULL;
    if (end == src)
        return NULL;
    if (!isfinite(val))
        return NULL;

    if (to)
        *to = val;

    return end;
}

static const char *next_double(const char *src, double *to)
{
    const char *end;
    double      val;

    end = src;
    errno = 0;
    val = strtod(src, (char **)&end);
    if (errno)
        return NULL;
    if (end == src)
        return NULL;
    if (!isfinite(val))
        return NULL;

    if (to)
        *to = val;

    return end;
}

int parse_double(const char *src, double *to)
{
    const char *end;
    double      val;

    if (!src || !*src)
        return -1;

    end = skip_whitespace(next_double(src, &val));
    if (!end || *end)
        return -1;
    if (!isfinite(val))
        return -1;

    if (to)
        *to = val;

    return 0;
}

int parse_double_fraction(const char *src, double *to)
{
    const char *end;
    double      val, n, d;

    if (!src || !*src)
        return -1;

    end = skip_whitespace(next_double(src, &n));
    if (!end)
        return -1;

    if (*end == '/' || *end == ':')
        end++;
    else
        return -1;

    end = skip_whitespace(next_double(end, &d));
    if (!end || *end)
        return -1;
    if (!d)
        return -1;

    val = n / d;
    if (!isfinite(val))
        return -1;

    if (to)
        *to = val;

    return 0;
}

int parse_float(const char *src, float *to)
{
    const char *end;
    float       val;

    if (!src || !*src)
        return -1;

    end = skip_whitespace(next_float(src, &val));
    if (!end || (*end != 'f' && *end != 'F'))
        return -1;
    if (!isfinite(val))
        return -1;

    if (to)
        *to = val;

    return 0;
}

int parse_float_fraction(const char *src, float *to)
{
    const char *end;
    float       val, n, d;
    int         f = 0;

    if (!src || !*src)
        return -1;

    end = skip_whitespace(next_float(src, &n));
    if (!end)
        return -1;

    if (*end == 'f' || *end == 'F') {
        end = skip_whitespace(end + 1);
        f = 1;
    }

    if (*end == '/' || *end == ':')
        end++;
    else
        return -1;

    end = skip_whitespace(next_float(end, &d));
    if (*end == 'f' || *end == 'F') {
        end = skip_whitespace(end + 1);
        f = 1;
    }
    if (!end || *end || !f)
        return -1;
    if (!d)
        return -1;

    val = n / d;
    if (!isfinite(val))
        return -1;

    if (to)
        *to = val;

    return 0;
}

static const char *float_string(float value)
{
    static char  buf[1024];
    int          len;

    len = snprintf(buf, sizeof buf, "%.32f", value);
    if (len < 1 || len >= (int)sizeof buf)
        return "(BUG!)";

    while (len > 1 && buf[len-1] != '.' && buf[len-2] != '.') {
        char   old = buf[--len];
        float  tmp;

        buf[len] = '\0';
        if (!next_float(buf, &tmp) || tmp != value) {
            buf[len] = old;
            break;
        }
    }

    return buf;
}

static const char *double_string(double value)
{
    static char  buf[1024];
    int          len;

    len = snprintf(buf, sizeof buf, "%.64f", value);
    if (len < 1 || len >= (int)sizeof buf)
        return "(BUG!)";

    while (len > 1 && buf[len - 1] != '.' && buf[len - 2] != '.') {
        char    old = buf[--len];
        double  tmp;

        buf[len] = '\0';
        if (!next_double(buf, &tmp) || tmp != value) {
            buf[len] = old;
            break;
        }
    }

    return buf;
}

int find_float_fraction(float value, int32_t *nto, uint32_t *dto)
{
    for (uint32_t  d = 1; d < 33554432; d++) {
        int32_t  n = (int32_t)((double)value * (double)d - 0.5);
        if ((float)((float)n / (float)d) == value) {
            if (nto)
                *nto = n;
            if (dto)
                *dto = d;
            return 0;
        }
        n++;
        if ((float)((float)n / (float)d) == value) {
            if (nto)
                *nto = n;
            if (dto)
                *dto = d;
            return 0;
        }
        n++;
        if ((float)((float)n / (float)d) == value) {
            if (nto)
                *nto = n;
            if (dto)
                *dto = d;
            return 0;
        }
    }
    return -1;
}

int find_double_fraction(double value, int32_t *nto, uint32_t *dto)
{
    for (uint32_t  d = 1; d < 33554432; d++) {
        int32_t  n = (int32_t)((double)value * (double)d - 0.5);
        if ((double)((double)n / (double)d) == value) {
            if (nto)
                *nto = n;
            if (dto)
                *dto = d;
            return 0;
        }
        n++;
        if ((double)((double)n / (double)d) == value) {
            if (nto)
                *nto = n;
            if (dto)
                *dto = d;
            return 0;
        }
        n++;
        if ((double)((double)n / (double)d) == value) {
            if (nto)
                *nto = n;
            if (dto)
                *dto = d;
            return 0;
        }
    }
    return -1;
}

void show32(const char *src, word32 w)
{
    int32_t   n;
    uint32_t  d;

    printf("%s\n", src);
    printf("    Hex: 0x%" PRIx32 "\n", w.u32);
    if (w.i32 < 0)
        printf("    Dec: %" PRIu32 " = %" PRIi32 "\n", w.u32, w.i32);
    else
        printf("    Dec: %" PRIu32 "\n", w.u32);
    printf("    Bin: 0b%s\n", u32_bits(w.u32));
    printf("  Float: %s = %s\n", float_bits(w.f), float_string(w.f));
    fflush(stdout);
    if (!find_float_fraction(w.f, &n, &d))
        printf("   Frac: %" PRIi32 " / %" PRIu32 "\n", n, d);
    printf("\n");
    fflush(stdout);
}

void show64(const char *src, word64 w)
{
    int32_t   n;
    uint32_t  d;

    printf("%s\n", src);
    printf("    Hex: 0x%" PRIx64 "\n", w.u64);
    if (w.i64 < 0)
        printf("    Dec: %" PRIu64 " = %" PRIi64 "\n", w.u64, w.i64);
    else
        printf("    Dec: %" PRIu64 "\n", w.u64);
    printf("    Bin: 0b%s\n", u64_bits(w.u64));
    printf(" Double: %s = %s\n", double_bits(w.d), double_string(w.d));
    fflush(stdout);
    if (!find_double_fraction(w.d, &n, &d))
        printf("   Frac: %" PRIi32 " / %" PRIu32 "\n", n, d);
    printf("\n");
    fflush(stdout);
}

int main(int argc, char *argv[])
{
    if (((word32){ .f = 992.0f / 19883.0f }).u32 != UINT32_C(0x3D4C5B6A))
        fprintf(stderr, "Warning: Invalid 32-bit byte order, or 'float' not IEEE 754-2008 Binary32.\n");
    if (((word64){ .d = -138242.0 / 65025.0 }).u64 != UINT64_C(0xc001020304050607))
        fprintf(stderr, "Warning: Invalid 64-bit byte order, or 'double' not IEEE 754-2008 Binary64.\n");

    if (argc < 2 || !strcmp(argv[1], "-h") || !strcmp(argv[1], "--help")) {
        const char *arg0 = (argc > 0 && argv && argv[0] && argv[0][0]) ? argv[0] : "(this)";
        fprintf(stderr, "\n");
        fprintf(stderr, "Usage: %s [ -h | --help ]\n", arg0);
        fprintf(stderr, "       %s VALUE [ VALUE ... ]\n", arg0);
        fprintf(stderr, "Where VALUE can be\n");
        fprintf(stderr, " - a 32-bit integer (in octal, decimal, or hexadecimal)\n");
        fprintf(stderr, " - a 32-bit floating point number (with 'f' suffix)\n");
        fprintf(stderr, " - a pair of 32-bit floating point numbers with / in between,\n");
        fprintf(stderr, "   to denote a fraction (at least one with 'f' suffix)\n");
        fprintf(stderr, " - a 64-bit integer (in octal, decimal, or hexadecimal)\n");
        fprintf(stderr, " - a 64-bit floating point number (without any suffix)\n");
        fprintf(stderr, " - a pair of 64-bit floating point numbers with / in between,\n");
        fprintf(stderr, "   to denote a fraction (neither having any suffix)\n");
        fprintf(stderr, "\n");
        return EXIT_SUCCESS;
    }

    for (int arg = 1; arg < argc; arg++) {
        uintmax_t  u;
        intmax_t   i;
        double     d;
        float      f;

        if (!parse_uintmax(argv[arg], &u)) {
            if ((uintmax_t)((uint32_t)u) == u) {
                show32(argv[arg], (word32){ .u32 = u });
                continue;
            } else
            if ((uintmax_t)((uint64_t)u) == u) {
                show64(argv[arg], (word64){ .u64 = u });
                continue;
            }
        }

        if (!parse_intmax(argv[arg], &i)) {
            if ((intmax_t)((int32_t)i) == i) {
                show32(argv[arg], (word32){ .i32 = i });
                continue;
            } else
            if ((intmax_t)((int64_t)i) == i) {
                show64(argv[arg], (word64){ .i64 = i });
                continue;
            }
        }

        if (!parse_float(argv[arg], &f) || !parse_float_fraction(argv[arg], &f)) {
            show32(argv[arg], (word32){ .f = f });
            continue;
        }

        if (!parse_double(argv[arg], &d) || !parse_double_fraction(argv[arg], &d)) {
            show64(argv[arg], (word64){ .d = d });
            continue;
        }

        fprintf(stderr, "%s: Cannot parse parameter.\n", argv[arg]);
        return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
}

It itself uses 992/19883 ≃ 0.049891866, which corresponds to 0x3d4c5b6a on architectures with the same byte order for floats and ints and floats are in IEEE 754-2008 Binary32 format; and -138242/65025 ≃ -2.125982314494425, which corresponds to 0xc001020304050607 on architectures with the same byte order for doubles and ints and doubles are in IEEE 754-2008 Binary64 format.

If there is a simple fraction to represent your chosen value, it finds and reports that too. Which is nice, because the division (of two integers) is often easier to specify the exact floating-point value; certain decimal floating-point numbers can have rounding issues. (Although they should always be a specific value, some (older or over-optimized) software may parse and round them incorrectly. The program assumes you use a trustworthy compiler and C library, as the floating-point value it shows is truncated to the minimum sufficient number of digits still parsing to the correct value.)

On Teensies, depending on your compiler options, 'double' may be the same type as 'float', but when it is not, it does correspond to IEEE 754-2008 Binary64; and floats do correspond to IEEE 754-2008 Binary32; and integers and floating-point numbers have the same byte order. Similarly for most ARMs running Linux, as well as on desktop Intel and AMD x86 and x86-64 processors.

I've used this technique for storing binary data in native format when throughput has been paramount (including HPC, in MD simulations; even from Fortran 95 code), and it works rather well. I usually augment the 20 bytes or so of header prototype values with some kind of 32-bit format identifier, rounding it to nice 24 bytes; add a 64-bit data size/length/count, and you're at 32 bytes.

In the case of connecting to a Teensy via USB Serial or similar, you'll probably use floats (and not doubles), so consider starting I/O by Teensy sending a prototype 32-bit value that also acts like a version number or "Hello, your software running on Teensy is running fine" message. If you end up having to change the protocol later, you can change that prototype, and have your application remember which version of protocol it should be using; that way the new application works with both old and new Teensy firmware. Which is nice.

 

[DavidBell]

I got the problem down to scanning large unsigned ints.

Program:

// Edit to add
    asm(".global _printf_float");

#include <Arduino.h>
#include <inttypes.h>

extern "C" int main(void)
    {
        delay(1000);

        Serial.begin(9600);

        uint32_t tHe = 0xBEAC0508, tPG = 0x3E5D0FA2;

        char d[128] = {0};

        Serial.println("Pre-scan");
        Serial.printf("%08lX,%08lX\n", tHe, tPG);
        Serial.printf("%08u,%08u\n", tHe, tPG);

        // print/scan
        sprintf(d, "%08lX,%08lX\n", tHe, tPG);
        tHe = 0; tPG = 0;
        sscanf(d, "%08lX,%08lX", &tHe, &tPG);


        Serial.println("\nPost-scan");
        Serial.printf("%08lX,%08lX\n", tHe, tPG);
        Serial.printf("%08u,%08u\n", tHe, tPG);
    }

Output:

Pre-scan
BEAC0508,3E5D0FA2
3198944520,1046286242

Post-scan
[B]7FFFFFFF[/B],3E5D0FA2
[B]2147483647[/B],1046286242

Note the number greater than 2^31 fails on the scan. I think 32nd bit is sign which explains why punning a negative float to uint32_t fails.

Have I got the wrong scan specifiers?

Note this is main() with a makefile. Also a more recent compiler than ships with teensyduino.

I can try on pure arduino setup later this evening.

 

[DavidBell]

With vanilla arduino ide the code above executes correctly. (gcc 5.4?)

But on my other set up that uses a makefile and arm-gcc 11.2 compiler I get the incorrect scan result.

Tomorrow I'll take my non-connected clean T36 to the other set up and see if I can run the arduino ide from there.

 

[MichaelMeissner]

I got the problem down to scanning large unsigned ints.

Program:

// Edit to add
    asm(".global _printf_float");

#include <Arduino.h>
#include <inttypes.h>

extern "C" int main(void)
    {
        delay(1000);

        Serial.begin(9600);

        uint32_t tHe = 0xBEAC0508, tPG = 0x3E5D0FA2;

        char d[128] = {0};

        Serial.println("Pre-scan");
        Serial.printf("%08lX,%08lX\n", tHe, tPG);
        Serial.printf("%08u,%08u\n", tHe, tPG);

        // print/scan
        sprintf(d, "%08lX,%08lX\n", tHe, tPG);
        tHe = 0; tPG = 0;
        sscanf(d, "%08lX,%08lX", &tHe, &tPG);


        Serial.println("\nPost-scan");
        Serial.printf("%08lX,%08lX\n", tHe, tPG);
        Serial.printf("%08u,%08u\n", tHe, tPG);
    }

Output:

Pre-scan
BEAC0508,3E5D0FA2
3198944520,1046286242

Post-scan
[B]7FFFFFFF[/B],3E5D0FA2
[B]2147483647[/B],1046286242

Note the number greater than 2^31 fails on the scan. I think 32nd bit is sign which explains why punning a negative float to uint32_t fails.

Have I got the wrong scan specifiers?

Yes.

The uint<size>_t and int<size>_t cannot be used with the scanf and printf % formats. The scanf/printf formats can only handle standard types. The <size> types are mapped into the appropriate standard type, which varies from machine to machine, from compiler to compiler, and it can change with different compiler options.

The ISO C/C++ standards defined a header file (inttypes.h) that provides format suffixes in full ISO environments, but whether or not the embedded compilers like used for Teensy or Arduino boards support this, might be an issue.

If you can't use the PRI... and SCN... macros, then you need to use the standard types, such as unsigned long. To printf the type, you can convert it on the printf call:

    char buffer[100];
    // ...
    sprintf (buffer, "Foo = %lx\n", (unsigned long)bar);

For scanf, you need a variable with a standard type that you pass to scanf and then store that variable into your appropriately sized variable:

    unsigned long tmp;
    char buffer[100];
    uint32_t var;
    // ...
    sscanf (buffer, "%lx", &tmp);
    var = tmp;

I'm going to edit your code to try and add the appropriate syntax, but I'm not going to try it out:

// Edit to add
    asm(".global _printf_float");

#include <Arduino.h>
#include <inttypes.h>

extern "C" int main(void)
    {
        delay(1000);

        Serial.begin(9600);

        uint32_t tHe = 0xBEAC0508, tPG = 0x3E5D0FA2;

        char d[128] = {0};

        Serial.println("Pre-scan");
        Serial.printf("%08" PRIx32 ",%08" PRIx32 "\n", tHe, tPG);
        Serial.printf("%08lu,%08lu\n", (unsigned long)tHe, (unsigned long)tPG);
 
        // print/scan
        sprintf(d, "%08" PRIx32 ",%08" PRIx32 "\n", tHe, tPG);
        tHe = 0; tPG = 0;
        sscanf(d, "%08" SCNx32 ",%08" SCNx32, &tHe, &tPG);


        Serial.println("\nPost-scan");
        Serial.printf("%08" PRIx32 ",%08" PRIx32 "\n", tHe, tPG);
        Serial.printf("%08lu,%08lu\n", (unsigned long)tHe, (unsigned long)tPG);
    }

Note this is main() with a makefile. Also a more recent compiler than ships with teensyduino.

I can try on pure arduino setup later this evening.

And it may or may not work, since many other Arduino systems don't support Serial.printf.

And for many of the older AVR boards, the whole printf and scanf libraries might take up all available flash memory space, particularly if the floating point options are enabled, which drags in the whole floating point software support. Note for Teensys if you compile for saving space, a version of the printf/scanf libraries is used that does not support the floating point formats is used. Similarly for other machines, often times the floating point formats are not used at all.

 

[DavidBell]

I did look up inttypes.h at the other site, and I'm pretty sure for SCNx32 it was "lx", but here it is "x".

In any case I'll try more options at both sites, including with the macros (thanks, you didn't need to do that), and report back.

I suspect I am going to change this to direct byte routines in the future, as scanf/uint##_t might remain a moving target. It was sort of on the list anyway.

 

[Nominal Animal]

The <inttypes.h> macros vary from OS to OS, and from architecture to architecture. SCNx32 in particular can be "lx" on ILP32 but "x" on all 64-bit data models (LLP64, LP64, and so on).

Similarly, SCNx64 can be "x" on ILP64, "lx" on LP64, and "llx" on LLP64.

You could use const char *hex_to_u32(const char *src, uint32_t *to) and const char *skip_whitespace(const char *src) from reply #7, so that to parse "<id> ':' <x> ',' <y>" for example,

const char *unpack(const char *src)
{
    uint32_t  id;
    word32  x, y;

    src = skip_whitespace(hex_to_u32(src, &id));
    if (!src || *src != ':')
        return NULL;
    else
        src = skip_whitespace(src + 1);

    src = skip_whitespace(hex_to_u32(src, &x.u32));
    if (!src || *src != ',')
        return NULL;
    else
        src = skip_whitespace(src + 1);

    src = hex_to_u32(src, &y.u32);
    if (!src)
        return NULL;

    /* Do something with id, x.u32 (or x.f), and y.u32 (or y.f) */

    return src;
}

That function takes a pointer to the string. If it is a buffer from USB Serial, remember that it must be terminated with a nul char, which isn't transferred (you need to add it yourself).
If it can parse the triplet, it returns a pointer to the first char following the triplet (which, if this is a full message or string, will be nul char; if full line, a newline).

The reverse formatting takes a buffer of at least 28 characters (8 + 1 + 8 + 1 + 8 + newline + terminating nul = 28):

int pack(char *buffer, uint32_t id, float x, float y)
{
    char *p = buffer;

    p = u32_to_hex(p, 8, id);
    *(p++) = ':';
    p = u32_to_hex(p, 8, float_to_u32(x));
    *(p++) = ',';
    p = u32_to_hex(p, 8, float_to_u32(y));
    *(p++) = '\n';
    *p = '\0';

    return (int)(uintptr_t)(p - buffer);
}

It returns 27, the number of characters in the buffer, excluding the terminating nul char, because that's the number of chars you need to transfer.
The newline at end is so that the receiver can read an entire 27-byte line: unpack(line) will return a pointer to a newline then, i.e.

    const char *end = unpack(src);
    if (!end || end[0] != '\n' || end[1] != '\0') {
        /* Failed, output an error, possibly the entire line 'src'. */
    } else {
        /* Successful */
    }

 

[DavidBell]

So more test results, using my code posted in #10 and MichaelMeissner's in #12

In Ardunio IDE,

"lX", "X", SCNx32 all work, though "lX" has wrong type warnings.

In my other set up using tytools commands & gcc 11.2

Only SCNx32 works, "X" gives type warnings

So it seems to be a compiler, or compiler switch thing.

As noted above, now I know where the problem is I'll probably switch to unioning structs with char[] and do hex conversions directly. I was sort of going to do that anyway, just the formatting makes things a bit easier while I'm working out the messaging.

Thanks for your help.