Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 4 of 4

Thread: Programming Teensy 4.1 from remote Balena container

  1. #1
    Junior Member
    Join Date
    Aug 2022
    Posts
    3

    Programming Teensy 4.1 from remote Balena container

    I am working on a project utilizing the Teensy 4.1 connected to a Raspberry Pi 4. The Raspberry Pi runs BalenaOS (https://www.balena.io/cloud/) and runs services in docker containers that are managed by the Balena cloud. For those of you not familiar Balena is a deploy system built on docker compose.

    A firm requirement of this project is that we will not be able to access the Teensy's push button (I know I know you've heard it before).

    One obvious mistake we are potentially making is that we do not currently use the UDEV rules, that is something I definitely intend to add ASAP but I'm pretty confused by some of the behavior I'm seeing and I was hoping I might gain some insight, as well as understanding regarding how the UDEV rules might prevent the issues I'm seeing (if at all) and if we need them installed in the container or the host or both.

    Our docker compose file looks like this:

    Code:
      workspace:
        build:
          context: .
          dockerfile: dockerfile-workspace.template
        restart: "no"
        depends_on:
          - watchdog
        privileged: true
        network_mode: host
    And then inside the container we run UDEV and use PlatformIO to develop the Teensy's firmware:

    Code:
    FROM balenalib/%%BALENA_MACHINE_NAME%%-python:3.9-bullseye
    
    RUN apt update && apt install -y \
        build-essential \
        cmake \
        libudev-dev \
        libusb-dev \
        qtbase5-dev \
        pkg-config \
        git \
        psmisc \
        vim \
        openssh-server
    
    # Enable udevd so that USB devices show up in our container
    ENV UDEV=on
    
    RUN pip install --upgrade pip
    
    WORKDIR /workspace
    
    # Install platformio
    RUN curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o get-platformio.py
    RUN python3 get-platformio.py
    RUN ln -s ~/.platformio/penv/bin/pio /usr/local/bin/pio || :
    RUN ln -s ~/.platformio/penv/bin/platformio /bin/platformio || :
    
    RUN mkdir /var/run/sshd 
    
    COPY ./microcontroller ./microcontroller
    RUN pio pkg install -d ./microcontroller
    
    COPY ./scripts/run/workspace.sh ./
    
    # Script that optionally starts an ssh server
    CMD [ "/workspace/workspace.sh" ]
    We use privileged mode instead of device mapping because of the Teensy's behavior as an HID device since, we found timing could be unpredictable on startup.

    Until now we have mostly been successful without adding UDEV rules. The Teensy shows up as /dev/ttyACM0 when it's been programmed, and when it is not we are able to flash it using the UART interface /dev/ttyAMA0. If it is being stubborn about going into HalfKay we are able to force it by setting the port to baudrate 134 (https://forum.pjrc.com/threads/63306...l=1#post253633).

    If there are issues with PlatformIO we are happy to use the teensy_loader_cli directly. Or I have even (after a recently catastrophic failure) tried installing and running the full Teensyduino GUI (https://www.balena.io/blog/running-a...h-balenacloud/).


    This has run mostly successfully for about a month or so, however for the second time I am seeing behavior that I find troubling. After a recent power cycle on two of my setups (after just having programmed the Teensy using PlatformIO) the devices did not show up as /dev/ttyACM0. When I try to program them (whether it be with PlatformIO or Teensyduino) the loader hangs with:

    Code:
    Waiting for Teensy device...
     (hint: press the reset button)
    At this point the Teensy will require running the LED Blink Restore program (which requires pushing the button)...however I need to prepare for the reality of these devices being in the "field" and getting to the button will be difficult. Having to take apart the device to get to the button will be regarding as a major failing for the requirements of this project. So in addition to learning why this happens, I'd like to see if there's some way (any way) to recover the Teensy remotely.

    I have tried:

    • Adding the UDEV rules to the container and host and restarting UDEV
    • Adding the UDEV rules to the container and host and power cycling the Raspberry Pi 4's USB hub using usbctl
    • Full power cycles of the device (I have a watchdog I can trip). Unfortunately this is yet to be done with the UDEV rules because Balena requires me to hack on the OS config to load them persistently startup and I'm not convinced enough that the power cycle is functionally different than power cycling the USB hub/restarting UDEV (however if anyone disagrees please let me know and eli5).


    Some more symptoms:

    • Nothing in dmesg when I power cycle the usb hub
    • No device shows up in lsusb


    It's as if the Teensy's are completely disconnected.

    Another weird observation that I believe to be unrelated but figured I'd mention is that with my current firmware version we have to always program the teensy twice. The first time always fails predictably with an error message:

    Code:
    error writing to Teensy
    
    *** [upload] Error 1
    And the second time is always successful.

    This is not true if we use a simple test sketch and my hunch is that some dependency we are pulling in is doing something weird.

    Our PlatformIO config:

    Code:
    [env:teensy]
    # Latest has bug fix
    platform = https://github.com/platformio/platform-teensy.git
    framework = arduino
    board = teensy41
    build_flags = -DUSB_SERIAL -DDEBUG 
    lib_deps = 
       arduino-libraries/ArduinoRS485
       SPI
       Wire
       adafruit/Adafruit Unified Sensor@^1.1.6
       adafruit/Adafruit BusIO@^1.13.2
       adafruit/Adafruit MAX31865 library@^1.5.0
       adafruit/Adafruit BME280 Library@^2.2.2
       mobizt/FirebaseJson@^3.0.0
    
    lib_ldf_mode = deep+
    upload_protocol = teensy-cli
    test_filter = test/*
    Headers included in the firmware:

    Code:
    #include <Arduino.h>
    #include <FirebaseJson.h>
    #include <ArduinoRS485.h>
    #include <Adafruit_MAX31865.h>
    #include <Adafruit_BME280.h>
    That said this failure happened after I had flashed a simple sketch onto the Teensy's for debugging. This sketch only includes <Arduino.h>, apologies for the messy code:

    Code:
    // Arduino sketch for initializing/playing with the vaisala sensors
    #include <Arduino.h>
    
    constexpr uint8_t PIN_TXENA0 = 37;
    constexpr uint8_t PIN_TXENA1 = 5;
    HardwareSerial *rs485Uart0 = &Serial3;
    HardwareSerial *rs485Uart1 = &Serial4;
    
    void xmitEna(int i, bool ena){
      if(i == 0){
        digitalWrite(PIN_TXENA0, ena ? HIGH : LOW);
      } else {
        digitalWrite(PIN_TXENA1, ena ? HIGH : LOW);
      }
    }
    
    void setup() {
      pinMode(PIN_TXENA0, OUTPUT);
      pinMode(PIN_TXENA1, OUTPUT);
      xmitEna(0, false);
      xmitEna(1, false);
      
      rs485Uart0->begin(19200);
      rs485Uart1->begin(19200);
      
      xmitEna(0, true);
      xmitEna(1, true);
      rs485Uart0->print("\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r");
      rs485Uart0->flush();
      xmitEna(0, false);
      
      rs485Uart1->print("\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r");
      rs485Uart1->flush();
      xmitEna(1, false);
      
      Serial.begin(9600); // USB Serial
      while(!Serial); // Wait for serial terminal to connect.  
      Serial.println("Vaisala Test");
      pinMode(4, OUTPUT);
    }
    
    
    unsigned int getCRC16(unsigned char *buf, int len) {
      unsigned int crc = 0xFFFF;
      for (int pos = 0; pos < len; pos++) {
        crc ^= (unsigned int)buf[pos]; // XOR byte into least sig. byte of crc
    
        for (int i = 8; i != 0; i--) { // Loop over each bit
          if ((crc & 0x0001) != 0) {   // If the LSB is set
            crc >>= 1;                 // Shift right and XOR 0xA001
            crc ^= 0xA001;
          } else       // Else LSB is not set
            crc >>= 1; // Just shift right
        }
      }
    
      // Swap bytes
      return (crc >> 8) + (crc << 8);
    }
    
    uint8_t deviceAddress = 241;
    int holdingRegisterRead(HardwareSerial * Interface, int idx, uint16_t address, uint16_t &value,
                                         String &errorMessage,
                                         size_t readWriteDelay, size_t timeout) {
      // Flush input buffer
      const size_t packetlen = 8;
      uint8_t packet[packetlen] = {};
      xmitEna(idx, true);
      delay(1000);
      packet[0] = deviceAddress; // Device address
      packet[1] = (uint8_t)0x03;
      packet[2] = (uint8_t)(address >> 8);
      packet[3] = (uint8_t)address; // Coil offset or address (16 bit)
      // This protocol can be used to read many input registers at once
      // but we are restricting it to one value at a time
      packet[4] = (uint8_t)0x00;
      packet[5] = (uint8_t)0x01;
      uint16_t crc = getCRC16(packet, 6);
      packet[6] = (uint8_t)(crc >> 8);
      packet[7] = (uint8_t)crc;
      Interface->write(packet[0]);
        Interface->flush();
      delay(5);
      Interface->write(packet[1]);
        Interface->flush();
        delay(5);
      Interface->write(packet[2]);
        delay(5);
      Interface->write(packet[3]);
        Interface->flush();
        delay(5);
      Interface->write(packet[4]);
        Interface->flush();
        delay(5);
      Interface->write(packet[5]);
        Interface->flush();
        delay(5);
      Interface->write(packet[6]);
        Interface->flush();
        delay(5);
      Interface->write(packet[7]);
        Interface->flush();
      delay(5);
      Interface->flush();
      delay(5);
      xmitEna(idx, false);
      delay(readWriteDelay);
      delay(500);
      uint8_t headerByte = 0xff;
      int i = 0;
      String failure = String(idx) + ": ";
      for (elapsedMillis timer = 0;
           timer < timeout && headerByte != deviceAddress;) {
            while(Interface->available()){
              headerByte = Interface->read();
              i++;
              if(headerByte == deviceAddress){
                break;
              }
              if(headerByte != 0xff){
                failure += "[" + String(headerByte, 16) + "]";
              }
            }
      }
      if (headerByte != deviceAddress) {
        Serial.printf("REad %i\n", i);
        Serial.println(failure);
        errorMessage = "Timed out waiting for header";
        return 0xFFFF;
      }
    
      uint8_t receivedFunctionCode = Interface->read();
      if (receivedFunctionCode != 0x03) {
        uint8_t exceptionCode = Interface->read();
        errorMessage = "Exception code: " + String(exceptionCode, 16);
        return exceptionCode;
      }
    
      uint8_t byteCount = Interface->read(); // register count * 2
      if (byteCount != 2) {
        errorMessage = "Unexpected byte count";
        return 0xFFFE;
      }
    
      const size_t responseLen = 7;
      uint8_t responsePacket[responseLen] = {};
      responsePacket[0] = headerByte;
      responsePacket[1] = receivedFunctionCode;
      responsePacket[2] = byteCount;
      // Value
      responsePacket[3] = Interface->read();
      responsePacket[4] = Interface->read();
      // CRC
      responsePacket[5] = Interface->read();
      responsePacket[6] = Interface->read();
    
      uint16_t receivedCRC = (responsePacket[5] << 8) + responsePacket[6];
    
      uint16_t expectedCRC = getCRC16(responsePacket, responseLen - 2);
      if (receivedCRC != expectedCRC) {
        errorMessage = "Received: ";
        for (size_t i = 0; i < responseLen; i++) {
          errorMessage += "[" + String(responsePacket[i], 16) + "]";
        }
        errorMessage += ", expected CRC: " + String(expectedCRC, 16);
        return 0xFFFA;
      }
      value = (responsePacket[3] << 8) + responsePacket[4];
      return 0;
    }
    
    void loop() {  
      /*
      while(rs485Uart0->available()){
        rs485Uart0->read();
      }
      while(rs485Uart1->available()){
        rs485Uart1->read();
      }
      */
      String command = "smode stop";
      xmitEna(0, true);
      rs485Uart0->print(command + "\r");
      rs485Uart0->flush();
      xmitEna(0, false);
      
    
      xmitEna(1, true);
      rs485Uart1->print(command + "\r");
      rs485Uart1->flush();
      xmitEna(1, false);
     
      delay(100);
    
      String response0;
      Serial.println("\r\n\n0 Response:");
      for (int i=0; i<100; i++){
        while(rs485Uart0->available()){
          char c = rs485Uart0->read();
          if(c != 0){
            response0 += c;
          }
        }
        delay(10);
      }
      Serial.println(response0);
      
      Serial.println("\r\n\n1 Response:");
      String response1 = "";
      for (int i=0; i<100; i++){
        while(rs485Uart1->available()){
          char c = rs485Uart1->read();
          if(c != 0){
            response1 += c;
          }
        }
        delay(10);
      }
      Serial.println(response1);
    
      
    /*
      uint16_t value;
      String errorMessage;
      int ret = holdingRegisterRead(rs485Uart0, 0, 0x0101, value,
                                         errorMessage,
                                         1000, 1000);
      Serial.printf("RET0 %i, %i\n", ret, value * 10);
      Serial.println(errorMessage);
      */
     
      // Flush input buffer
    }
    If anyone can share any insight or point out all the wrong assumptions I'm making that would be super helpful. Also I was wondering if it might be possible to trigger the button press from the Raspberry Pi somehow...or if there's a remote way to trigger the LED Blink Restore? Could the buttons contacts be wired to a GPIO pin or something?? That'd be the ultimate fail-safe.
    Last edited by loweja01; 11-12-2022 at 04:00 PM. Reason: Added context

  2. #2
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    27,085
    Programming Teensy always uses HID protocol. Serial is never used. The 134 baud rate command causes Teensy to reboot into bootloader mode, which is HID protocol. It is never USB Serial during programming. It only becomes USB Serial again after rebooting to the program you've just written.

    If loading new code fails, Teensy can be left without a program which communicates on the USB port. In that case, it can never hear the 134 baud rate message or anything else to tell it to go into bootloader mode. Likewise if a buggy program is ever loaded which crashes in any way that disrupts USB communication. Not even power cycling will recover, because it will just try again to run whatever partial program was put into the flash memory. This is the reason why every Teensy is made with a pushbutton, so you can recover.

    Sounds like you should focus on that "error writing to Teensy", as a failed upload will leave it without a valid program.

    Ultimately the only 100% reliable way to recover from errors will look like wiring one of the Pi's GPIO pins to the Program pin, or a transistor which pulls the Program pin low.

    But hopefully you'll somehow manage to figure out why the communication is getting errors during code upload. This container situation and lack of udev rules is pretty far outside of anything I can directly support. But hopefully this limited info helps.

  3. #3
    Junior Member
    Join Date
    Aug 2022
    Posts
    3
    Quote Originally Posted by PaulStoffregen View Post
    Programming Teensy always uses HID protocol. Serial is never used. The 134 baud rate command causes Teensy to reboot into bootloader mode, which is HID protocol. It is never USB Serial during programming. It only becomes USB Serial again after rebooting to the program you've just written.

    If loading new code fails, Teensy can be left without a program which communicates on the USB port. In that case, it can never hear the 134 baud rate message or anything else to tell it to go into bootloader mode. Likewise if a buggy program is ever loaded which crashes in any way that disrupts USB communication. Not even power cycling will recover, because it will just try again to run whatever partial program was put into the flash memory. This is the reason why every Teensy is made with a pushbutton, so you can recover.

    Sounds like you should focus on that "error writing to Teensy", as a failed upload will leave it without a valid program.

    Ultimately the only 100% reliable way to recover from errors will look like wiring one of the Pi's GPIO pins to the Program pin, or a transistor which pulls the Program pin low.

    But hopefully you'll somehow manage to figure out why the communication is getting errors during code upload. This container situation and lack of udev rules is pretty far outside of anything I can directly support. But hopefully this limited info helps.
    That is very helpful thank you! I like the the idea of wiring the Pi's GPIO pin as a fail-safe, I appreciate the confirmation that this is the only reliable way to ensure recoverability.

  4. #4
    Junior Member
    Join Date
    Aug 2022
    Posts
    3
    Just to follow up on this regarding the error
    Code:
    error writing to Teensy
    I've noticed that this error goes away when I comment out some of my code until the HEX file decreases in size to under about 132000 bytes (about 6.5% total flash usage).

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •