Programming Teensy 4.1 from remote Balena container

L

loweja01

New member
I am working on a project utilizing the Teensy 4.1 connected to a Raspberry Pi 4. The Raspberry Pi runs BalenaOS (https://www.balena.io/cloud/) and runs services in docker containers that are managed by the Balena cloud. For those of you not familiar Balena is a deploy system built on docker compose.

A firm requirement of this project is that we will not be able to access the Teensy's push button (I know I know you've heard it before).

One obvious mistake we are potentially making is that we do not currently use the UDEV rules, that is something I definitely intend to add ASAP but I'm pretty confused by some of the behavior I'm seeing and I was hoping I might gain some insight, as well as understanding regarding how the UDEV rules might prevent the issues I'm seeing (if at all) and if we need them installed in the container or the host or both.

Our docker compose file looks like this:

Code: 
  workspace:
    build:
      context: .
      dockerfile: dockerfile-workspace.template
    restart: "no"
    depends_on:
      - watchdog
    privileged: true
    network_mode: host

And then inside the container we run UDEV and use PlatformIO to develop the Teensy's firmware:

Code: 
FROM balenalib/%%BALENA_MACHINE_NAME%%-python:3.9-bullseye

RUN apt update && apt install -y \
    build-essential \
    cmake \
    libudev-dev \
    libusb-dev \
    qtbase5-dev \
    pkg-config \
    git \
    psmisc \
    vim \
    openssh-server

# Enable udevd so that USB devices show up in our container
ENV UDEV=on

RUN pip install --upgrade pip

WORKDIR /workspace

# Install platformio
RUN curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o get-platformio.py
RUN python3 get-platformio.py
RUN ln -s ~/.platformio/penv/bin/pio /usr/local/bin/pio || :
RUN ln -s ~/.platformio/penv/bin/platformio /bin/platformio || :

RUN mkdir /var/run/sshd 

COPY ./microcontroller ./microcontroller
RUN pio pkg install -d ./microcontroller

COPY ./scripts/run/workspace.sh ./

# Script that optionally starts an ssh server
CMD [ "/workspace/workspace.sh" ]

We use privileged mode instead of device mapping because of the Teensy's behavior as an HID device since, we found timing could be unpredictable on startup.

Until now we have mostly been successful without adding UDEV rules. The Teensy shows up as /dev/ttyACM0 when it's been programmed, and when it is not we are able to flash it using the UART interface /dev/ttyAMA0. If it is being stubborn about going into HalfKay we are able to force it by setting the port to baudrate 134 (https://forum.pjrc.com/threads/6330...ing-PlatformIO?p=253633&viewfull=1#post253633).

If there are issues with PlatformIO we are happy to use the teensy_loader_cli directly. Or I have even (after a recently catastrophic failure) tried installing and running the full Teensyduino GUI (https://www.balena.io/blog/running-a-gui-application-with-balenacloud/).


This has run mostly successfully for about a month or so, however for the second time I am seeing behavior that I find troubling. After a recent power cycle on two of my setups (after just having programmed the Teensy using PlatformIO) the devices did not show up as /dev/ttyACM0. When I try to program them (whether it be with PlatformIO or Teensyduino) the loader hangs with:

Code: 
Waiting for Teensy device...
 (hint: press the reset button)

At this point the Teensy will require running the LED Blink Restore program (which requires pushing the button)...however I need to prepare for the reality of these devices being in the "field" and getting to the button will be difficult. Having to take apart the device to get to the button will be regarding as a major failing for the requirements of this project. So in addition to learning why this happens, I'd like to see if there's some way (any way) to recover the Teensy remotely.

I have tried:

  • Adding the UDEV rules to the container and host and restarting UDEV
  • Adding the UDEV rules to the container and host and power cycling the Raspberry Pi 4's USB hub using usbctl
  • Full power cycles of the device (I have a watchdog I can trip). Unfortunately this is yet to be done with the UDEV rules because Balena requires me to hack on the OS config to load them persistently startup and I'm not convinced enough that the power cycle is functionally different than power cycling the USB hub/restarting UDEV (however if anyone disagrees please let me know and eli5).

Some more symptoms:

  • Nothing in dmesg when I power cycle the usb hub
  • No device shows up in lsusb

It's as if the Teensy's are completely disconnected.

Another weird observation that I believe to be unrelated but figured I'd mention is that with my current firmware version we have to always program the teensy twice. The first time always fails predictably with an error message:

Code: 
error writing to Teensy

*** [upload] Error 1

And the second time is always successful.

This is not true if we use a simple test sketch and my hunch is that some dependency we are pulling in is doing something weird.

Our PlatformIO config:

Code: 
[env:teensy]
# Latest has bug fix
platform = https://github.com/platformio/platform-teensy.git
framework = arduino
board = teensy41
build_flags = -DUSB_SERIAL -DDEBUG 
lib_deps = 
   arduino-libraries/ArduinoRS485
   SPI
   Wire
   adafruit/Adafruit Unified Sensor@^1.1.6
   adafruit/Adafruit BusIO@^1.13.2
   adafruit/Adafruit MAX31865 library@^1.5.0
   adafruit/Adafruit BME280 Library@^2.2.2
   mobizt/FirebaseJson@^3.0.0

lib_ldf_mode = deep+
upload_protocol = teensy-cli
test_filter = test/*

Headers included in the firmware:

Code: 
#include <Arduino.h>
#include <FirebaseJson.h>
#include <ArduinoRS485.h>
#include <Adafruit_MAX31865.h>
#include <Adafruit_BME280.h>

That said this failure happened after I had flashed a simple sketch onto the Teensy's for debugging. This sketch only includes <Arduino.h>, apologies for the messy code:

Code: 
// Arduino sketch for initializing/playing with the vaisala sensors
#include <Arduino.h>

constexpr uint8_t PIN_TXENA0 = 37;
constexpr uint8_t PIN_TXENA1 = 5;
HardwareSerial *rs485Uart0 = &Serial3;
HardwareSerial *rs485Uart1 = &Serial4;

void xmitEna(int i, bool ena){
  if(i == 0){
    digitalWrite(PIN_TXENA0, ena ? HIGH : LOW);
  } else {
    digitalWrite(PIN_TXENA1, ena ? HIGH : LOW);
  }
}

void setup() {
  pinMode(PIN_TXENA0, OUTPUT);
  pinMode(PIN_TXENA1, OUTPUT);
  xmitEna(0, false);
  xmitEna(1, false);
  
  rs485Uart0->begin(19200);
  rs485Uart1->begin(19200);
  
  xmitEna(0, true);
  xmitEna(1, true);
  rs485Uart0->print("\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r");
  rs485Uart0->flush();
  xmitEna(0, false);
  
  rs485Uart1->print("\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r");
  rs485Uart1->flush();
  xmitEna(1, false);
  
  Serial.begin(9600); // USB Serial
  while(!Serial); // Wait for serial terminal to connect.  
  Serial.println("Vaisala Test");
  pinMode(4, OUTPUT);
}


unsigned int getCRC16(unsigned char *buf, int len) {
  unsigned int crc = 0xFFFF;
  for (int pos = 0; pos < len; pos++) {
    crc ^= (unsigned int)buf[pos]; // XOR byte into least sig. byte of crc

    for (int i = 8; i != 0; i--) { // Loop over each bit
      if ((crc & 0x0001) != 0) {   // If the LSB is set
        crc >>= 1;                 // Shift right and XOR 0xA001
        crc ^= 0xA001;
      } else       // Else LSB is not set
        crc >>= 1; // Just shift right
    }
  }

  // Swap bytes
  return (crc >> 8) + (crc << 8);
}

uint8_t deviceAddress = 241;
int holdingRegisterRead(HardwareSerial * Interface, int idx, uint16_t address, uint16_t &value,
                                     String &errorMessage,
                                     size_t readWriteDelay, size_t timeout) {
  // Flush input buffer
  const size_t packetlen = 8;
  uint8_t packet[packetlen] = {};
  xmitEna(idx, true);
  delay(1000);
  packet[0] = deviceAddress; // Device address
  packet[1] = (uint8_t)0x03;
  packet[2] = (uint8_t)(address >> 8);
  packet[3] = (uint8_t)address; // Coil offset or address (16 bit)
  // This protocol can be used to read many input registers at once
  // but we are restricting it to one value at a time
  packet[4] = (uint8_t)0x00;
  packet[5] = (uint8_t)0x01;
  uint16_t crc = getCRC16(packet, 6);
  packet[6] = (uint8_t)(crc >> 8);
  packet[7] = (uint8_t)crc;
  Interface->write(packet[0]);
    Interface->flush();
  delay(5);
  Interface->write(packet[1]);
    Interface->flush();
    delay(5);
  Interface->write(packet[2]);
    delay(5);
  Interface->write(packet[3]);
    Interface->flush();
    delay(5);
  Interface->write(packet[4]);
    Interface->flush();
    delay(5);
  Interface->write(packet[5]);
    Interface->flush();
    delay(5);
  Interface->write(packet[6]);
    Interface->flush();
    delay(5);
  Interface->write(packet[7]);
    Interface->flush();
  delay(5);
  Interface->flush();
  delay(5);
  xmitEna(idx, false);
  delay(readWriteDelay);
  delay(500);
  uint8_t headerByte = 0xff;
  int i = 0;
  String failure = String(idx) + ": ";
  for (elapsedMillis timer = 0;
       timer < timeout && headerByte != deviceAddress;) {
        while(Interface->available()){
          headerByte = Interface->read();
          i++;
          if(headerByte == deviceAddress){
            break;
          }
          if(headerByte != 0xff){
            failure += "[" + String(headerByte, 16) + "]";
          }
        }
  }
  if (headerByte != deviceAddress) {
    Serial.printf("REad %i\n", i);
    Serial.println(failure);
    errorMessage = "Timed out waiting for header";
    return 0xFFFF;
  }

  uint8_t receivedFunctionCode = Interface->read();
  if (receivedFunctionCode != 0x03) {
    uint8_t exceptionCode = Interface->read();
    errorMessage = "Exception code: " + String(exceptionCode, 16);
    return exceptionCode;
  }

  uint8_t byteCount = Interface->read(); // register count * 2
  if (byteCount != 2) {
    errorMessage = "Unexpected byte count";
    return 0xFFFE;
  }

  const size_t responseLen = 7;
  uint8_t responsePacket[responseLen] = {};
  responsePacket[0] = headerByte;
  responsePacket[1] = receivedFunctionCode;
  responsePacket[2] = byteCount;
  // Value
  responsePacket[3] = Interface->read();
  responsePacket[4] = Interface->read();
  // CRC
  responsePacket[5] = Interface->read();
  responsePacket[6] = Interface->read();

  uint16_t receivedCRC = (responsePacket[5] << 8) + responsePacket[6];

  uint16_t expectedCRC = getCRC16(responsePacket, responseLen - 2);
  if (receivedCRC != expectedCRC) {
    errorMessage = "Received: ";
    for (size_t i = 0; i < responseLen; i++) {
      errorMessage += "[" + String(responsePacket[i], 16) + "]";
    }
    errorMessage += ", expected CRC: " + String(expectedCRC, 16);
    return 0xFFFA;
  }
  value = (responsePacket[3] << 8) + responsePacket[4];
  return 0;
}

void loop() {  
  /*
  while(rs485Uart0->available()){
    rs485Uart0->read();
  }
  while(rs485Uart1->available()){
    rs485Uart1->read();
  }
  */
  String command = "smode stop";
  xmitEna(0, true);
  rs485Uart0->print(command + "\r");
  rs485Uart0->flush();
  xmitEna(0, false);
  

  xmitEna(1, true);
  rs485Uart1->print(command + "\r");
  rs485Uart1->flush();
  xmitEna(1, false);
 
  delay(100);

  String response0;
  Serial.println("\r\n\n0 Response:");
  for (int i=0; i<100; i++){
    while(rs485Uart0->available()){
      char c = rs485Uart0->read();
      if(c != 0){
        response0 += c;
      }
    }
    delay(10);
  }
  Serial.println(response0);
  
  Serial.println("\r\n\n1 Response:");
  String response1 = "";
  for (int i=0; i<100; i++){
    while(rs485Uart1->available()){
      char c = rs485Uart1->read();
      if(c != 0){
        response1 += c;
      }
    }
    delay(10);
  }
  Serial.println(response1);

  
/*
  uint16_t value;
  String errorMessage;
  int ret = holdingRegisterRead(rs485Uart0, 0, 0x0101, value,
                                     errorMessage,
                                     1000, 1000);
  Serial.printf("RET0 %i, %i\n", ret, value * 10);
  Serial.println(errorMessage);
  */
 
  // Flush input buffer
}

If anyone can share any insight or point out all the wrong assumptions I'm making that would be super helpful. Also I was wondering if it might be possible to trigger the button press from the Raspberry Pi somehow...or if there's a remote way to trigger the LED Blink Restore? Could the buttons contacts be wired to a GPIO pin or something?? That'd be the ultimate fail-safe.
 
Last edited:
Programming Teensy always uses HID protocol. Serial is never used. The 134 baud rate command causes Teensy to reboot into bootloader mode, which is HID protocol. It is never USB Serial during programming. It only becomes USB Serial again after rebooting to the program you've just written.

If loading new code fails, Teensy can be left without a program which communicates on the USB port. In that case, it can never hear the 134 baud rate message or anything else to tell it to go into bootloader mode. Likewise if a buggy program is ever loaded which crashes in any way that disrupts USB communication. Not even power cycling will recover, because it will just try again to run whatever partial program was put into the flash memory. This is the reason why every Teensy is made with a pushbutton, so you can recover.

Sounds like you should focus on that "error writing to Teensy", as a failed upload will leave it without a valid program.

Ultimately the only 100% reliable way to recover from errors will look like wiring one of the Pi's GPIO pins to the Program pin, or a transistor which pulls the Program pin low.

But hopefully you'll somehow manage to figure out why the communication is getting errors during code upload. This container situation and lack of udev rules is pretty far outside of anything I can directly support. But hopefully this limited info helps.
 
PaulStoffregen said:
Programming Teensy always uses HID protocol. Serial is never used. The 134 baud rate command causes Teensy to reboot into bootloader mode, which is HID protocol. It is never USB Serial during programming. It only becomes USB Serial again after rebooting to the program you've just written.

If loading new code fails, Teensy can be left without a program which communicates on the USB port. In that case, it can never hear the 134 baud rate message or anything else to tell it to go into bootloader mode. Likewise if a buggy program is ever loaded which crashes in any way that disrupts USB communication. Not even power cycling will recover, because it will just try again to run whatever partial program was put into the flash memory. This is the reason why every Teensy is made with a pushbutton, so you can recover.

Sounds like you should focus on that "error writing to Teensy", as a failed upload will leave it without a valid program.

Ultimately the only 100% reliable way to recover from errors will look like wiring one of the Pi's GPIO pins to the Program pin, or a transistor which pulls the Program pin low.

But hopefully you'll somehow manage to figure out why the communication is getting errors during code upload. This container situation and lack of udev rules is pretty far outside of anything I can directly support. But hopefully this limited info helps.
Click to expand...

That is very helpful thank you! I like the the idea of wiring the Pi's GPIO pin as a fail-safe, I appreciate the confirmation that this is the only reliable way to ensure recoverability.
 
Just to follow up on this regarding the error
Code: 
error writing to Teensy
I've noticed that this error goes away when I comment out some of my code until the HEX file decreases in size to under about 132000 bytes (about 6.5% total flash usage).
 
You must log in or register to reply here.
Back
Top