I am working on a project utilizing the Teensy 4.1 connected to a Raspberry Pi 4. The Raspberry Pi runs BalenaOS (https://www.balena.io/cloud/) and runs services in docker containers that are managed by the Balena cloud. For those of you not familiar Balena is a deploy system built on docker compose.
A firm requirement of this project is that we will not be able to access the Teensy's push button (I know I know you've heard it before).
One obvious mistake we are potentially making is that we do not currently use the UDEV rules, that is something I definitely intend to add ASAP but I'm pretty confused by some of the behavior I'm seeing and I was hoping I might gain some insight, as well as understanding regarding how the UDEV rules might prevent the issues I'm seeing (if at all) and if we need them installed in the container or the host or both.
Our docker compose file looks like this:
And then inside the container we run UDEV and use PlatformIO to develop the Teensy's firmware:
We use privileged mode instead of device mapping because of the Teensy's behavior as an HID device since, we found timing could be unpredictable on startup.
Until now we have mostly been successful without adding UDEV rules. The Teensy shows up as /dev/ttyACM0 when it's been programmed, and when it is not we are able to flash it using the UART interface /dev/ttyAMA0. If it is being stubborn about going into HalfKay we are able to force it by setting the port to baudrate 134 (https://forum.pjrc.com/threads/6330...ing-PlatformIO?p=253633&viewfull=1#post253633).
If there are issues with PlatformIO we are happy to use the teensy_loader_cli directly. Or I have even (after a recently catastrophic failure) tried installing and running the full Teensyduino GUI (https://www.balena.io/blog/running-a-gui-application-with-balenacloud/).
This has run mostly successfully for about a month or so, however for the second time I am seeing behavior that I find troubling. After a recent power cycle on two of my setups (after just having programmed the Teensy using PlatformIO) the devices did not show up as /dev/ttyACM0. When I try to program them (whether it be with PlatformIO or Teensyduino) the loader hangs with:
At this point the Teensy will require running the LED Blink Restore program (which requires pushing the button)...however I need to prepare for the reality of these devices being in the "field" and getting to the button will be difficult. Having to take apart the device to get to the button will be regarding as a major failing for the requirements of this project. So in addition to learning why this happens, I'd like to see if there's some way (any way) to recover the Teensy remotely.
I have tried:
Some more symptoms:
It's as if the Teensy's are completely disconnected.
Another weird observation that I believe to be unrelated but figured I'd mention is that with my current firmware version we have to always program the teensy twice. The first time always fails predictably with an error message:
And the second time is always successful.
This is not true if we use a simple test sketch and my hunch is that some dependency we are pulling in is doing something weird.
Our PlatformIO config:
Headers included in the firmware:
That said this failure happened after I had flashed a simple sketch onto the Teensy's for debugging. This sketch only includes <Arduino.h>, apologies for the messy code:
If anyone can share any insight or point out all the wrong assumptions I'm making that would be super helpful. Also I was wondering if it might be possible to trigger the button press from the Raspberry Pi somehow...or if there's a remote way to trigger the LED Blink Restore? Could the buttons contacts be wired to a GPIO pin or something?? That'd be the ultimate fail-safe.
A firm requirement of this project is that we will not be able to access the Teensy's push button (I know I know you've heard it before).
One obvious mistake we are potentially making is that we do not currently use the UDEV rules, that is something I definitely intend to add ASAP but I'm pretty confused by some of the behavior I'm seeing and I was hoping I might gain some insight, as well as understanding regarding how the UDEV rules might prevent the issues I'm seeing (if at all) and if we need them installed in the container or the host or both.
Our docker compose file looks like this:
Code:
workspace:
build:
context: .
dockerfile: dockerfile-workspace.template
restart: "no"
depends_on:
- watchdog
privileged: true
network_mode: host
And then inside the container we run UDEV and use PlatformIO to develop the Teensy's firmware:
Code:
FROM balenalib/%%BALENA_MACHINE_NAME%%-python:3.9-bullseye
RUN apt update && apt install -y \
build-essential \
cmake \
libudev-dev \
libusb-dev \
qtbase5-dev \
pkg-config \
git \
psmisc \
vim \
openssh-server
# Enable udevd so that USB devices show up in our container
ENV UDEV=on
RUN pip install --upgrade pip
WORKDIR /workspace
# Install platformio
RUN curl -fsSL https://raw.githubusercontent.com/platformio/platformio-core-installer/master/get-platformio.py -o get-platformio.py
RUN python3 get-platformio.py
RUN ln -s ~/.platformio/penv/bin/pio /usr/local/bin/pio || :
RUN ln -s ~/.platformio/penv/bin/platformio /bin/platformio || :
RUN mkdir /var/run/sshd
COPY ./microcontroller ./microcontroller
RUN pio pkg install -d ./microcontroller
COPY ./scripts/run/workspace.sh ./
# Script that optionally starts an ssh server
CMD [ "/workspace/workspace.sh" ]
We use privileged mode instead of device mapping because of the Teensy's behavior as an HID device since, we found timing could be unpredictable on startup.
Until now we have mostly been successful without adding UDEV rules. The Teensy shows up as /dev/ttyACM0 when it's been programmed, and when it is not we are able to flash it using the UART interface /dev/ttyAMA0. If it is being stubborn about going into HalfKay we are able to force it by setting the port to baudrate 134 (https://forum.pjrc.com/threads/6330...ing-PlatformIO?p=253633&viewfull=1#post253633).
If there are issues with PlatformIO we are happy to use the teensy_loader_cli directly. Or I have even (after a recently catastrophic failure) tried installing and running the full Teensyduino GUI (https://www.balena.io/blog/running-a-gui-application-with-balenacloud/).
This has run mostly successfully for about a month or so, however for the second time I am seeing behavior that I find troubling. After a recent power cycle on two of my setups (after just having programmed the Teensy using PlatformIO) the devices did not show up as /dev/ttyACM0. When I try to program them (whether it be with PlatformIO or Teensyduino) the loader hangs with:
Code:
Waiting for Teensy device...
(hint: press the reset button)
At this point the Teensy will require running the LED Blink Restore program (which requires pushing the button)...however I need to prepare for the reality of these devices being in the "field" and getting to the button will be difficult. Having to take apart the device to get to the button will be regarding as a major failing for the requirements of this project. So in addition to learning why this happens, I'd like to see if there's some way (any way) to recover the Teensy remotely.
I have tried:
- Adding the UDEV rules to the container and host and restarting UDEV
- Adding the UDEV rules to the container and host and power cycling the Raspberry Pi 4's USB hub using usbctl
- Full power cycles of the device (I have a watchdog I can trip). Unfortunately this is yet to be done with the UDEV rules because Balena requires me to hack on the OS config to load them persistently startup and I'm not convinced enough that the power cycle is functionally different than power cycling the USB hub/restarting UDEV (however if anyone disagrees please let me know and eli5).
Some more symptoms:
- Nothing in dmesg when I power cycle the usb hub
- No device shows up in lsusb
It's as if the Teensy's are completely disconnected.
Another weird observation that I believe to be unrelated but figured I'd mention is that with my current firmware version we have to always program the teensy twice. The first time always fails predictably with an error message:
Code:
error writing to Teensy
*** [upload] Error 1
And the second time is always successful.
This is not true if we use a simple test sketch and my hunch is that some dependency we are pulling in is doing something weird.
Our PlatformIO config:
Code:
[env:teensy]
# Latest has bug fix
platform = https://github.com/platformio/platform-teensy.git
framework = arduino
board = teensy41
build_flags = -DUSB_SERIAL -DDEBUG
lib_deps =
arduino-libraries/ArduinoRS485
SPI
Wire
adafruit/Adafruit Unified Sensor@^1.1.6
adafruit/Adafruit BusIO@^1.13.2
adafruit/Adafruit MAX31865 library@^1.5.0
adafruit/Adafruit BME280 Library@^2.2.2
mobizt/FirebaseJson@^3.0.0
lib_ldf_mode = deep+
upload_protocol = teensy-cli
test_filter = test/*
Headers included in the firmware:
Code:
#include <Arduino.h>
#include <FirebaseJson.h>
#include <ArduinoRS485.h>
#include <Adafruit_MAX31865.h>
#include <Adafruit_BME280.h>
That said this failure happened after I had flashed a simple sketch onto the Teensy's for debugging. This sketch only includes <Arduino.h>, apologies for the messy code:
Code:
// Arduino sketch for initializing/playing with the vaisala sensors
#include <Arduino.h>
constexpr uint8_t PIN_TXENA0 = 37;
constexpr uint8_t PIN_TXENA1 = 5;
HardwareSerial *rs485Uart0 = &Serial3;
HardwareSerial *rs485Uart1 = &Serial4;
void xmitEna(int i, bool ena){
if(i == 0){
digitalWrite(PIN_TXENA0, ena ? HIGH : LOW);
} else {
digitalWrite(PIN_TXENA1, ena ? HIGH : LOW);
}
}
void setup() {
pinMode(PIN_TXENA0, OUTPUT);
pinMode(PIN_TXENA1, OUTPUT);
xmitEna(0, false);
xmitEna(1, false);
rs485Uart0->begin(19200);
rs485Uart1->begin(19200);
xmitEna(0, true);
xmitEna(1, true);
rs485Uart0->print("\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r");
rs485Uart0->flush();
xmitEna(0, false);
rs485Uart1->print("\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r\r");
rs485Uart1->flush();
xmitEna(1, false);
Serial.begin(9600); // USB Serial
while(!Serial); // Wait for serial terminal to connect.
Serial.println("Vaisala Test");
pinMode(4, OUTPUT);
}
unsigned int getCRC16(unsigned char *buf, int len) {
unsigned int crc = 0xFFFF;
for (int pos = 0; pos < len; pos++) {
crc ^= (unsigned int)buf[pos]; // XOR byte into least sig. byte of crc
for (int i = 8; i != 0; i--) { // Loop over each bit
if ((crc & 0x0001) != 0) { // If the LSB is set
crc >>= 1; // Shift right and XOR 0xA001
crc ^= 0xA001;
} else // Else LSB is not set
crc >>= 1; // Just shift right
}
}
// Swap bytes
return (crc >> 8) + (crc << 8);
}
uint8_t deviceAddress = 241;
int holdingRegisterRead(HardwareSerial * Interface, int idx, uint16_t address, uint16_t &value,
String &errorMessage,
size_t readWriteDelay, size_t timeout) {
// Flush input buffer
const size_t packetlen = 8;
uint8_t packet[packetlen] = {};
xmitEna(idx, true);
delay(1000);
packet[0] = deviceAddress; // Device address
packet[1] = (uint8_t)0x03;
packet[2] = (uint8_t)(address >> 8);
packet[3] = (uint8_t)address; // Coil offset or address (16 bit)
// This protocol can be used to read many input registers at once
// but we are restricting it to one value at a time
packet[4] = (uint8_t)0x00;
packet[5] = (uint8_t)0x01;
uint16_t crc = getCRC16(packet, 6);
packet[6] = (uint8_t)(crc >> 8);
packet[7] = (uint8_t)crc;
Interface->write(packet[0]);
Interface->flush();
delay(5);
Interface->write(packet[1]);
Interface->flush();
delay(5);
Interface->write(packet[2]);
delay(5);
Interface->write(packet[3]);
Interface->flush();
delay(5);
Interface->write(packet[4]);
Interface->flush();
delay(5);
Interface->write(packet[5]);
Interface->flush();
delay(5);
Interface->write(packet[6]);
Interface->flush();
delay(5);
Interface->write(packet[7]);
Interface->flush();
delay(5);
Interface->flush();
delay(5);
xmitEna(idx, false);
delay(readWriteDelay);
delay(500);
uint8_t headerByte = 0xff;
int i = 0;
String failure = String(idx) + ": ";
for (elapsedMillis timer = 0;
timer < timeout && headerByte != deviceAddress;) {
while(Interface->available()){
headerByte = Interface->read();
i++;
if(headerByte == deviceAddress){
break;
}
if(headerByte != 0xff){
failure += "[" + String(headerByte, 16) + "]";
}
}
}
if (headerByte != deviceAddress) {
Serial.printf("REad %i\n", i);
Serial.println(failure);
errorMessage = "Timed out waiting for header";
return 0xFFFF;
}
uint8_t receivedFunctionCode = Interface->read();
if (receivedFunctionCode != 0x03) {
uint8_t exceptionCode = Interface->read();
errorMessage = "Exception code: " + String(exceptionCode, 16);
return exceptionCode;
}
uint8_t byteCount = Interface->read(); // register count * 2
if (byteCount != 2) {
errorMessage = "Unexpected byte count";
return 0xFFFE;
}
const size_t responseLen = 7;
uint8_t responsePacket[responseLen] = {};
responsePacket[0] = headerByte;
responsePacket[1] = receivedFunctionCode;
responsePacket[2] = byteCount;
// Value
responsePacket[3] = Interface->read();
responsePacket[4] = Interface->read();
// CRC
responsePacket[5] = Interface->read();
responsePacket[6] = Interface->read();
uint16_t receivedCRC = (responsePacket[5] << 8) + responsePacket[6];
uint16_t expectedCRC = getCRC16(responsePacket, responseLen - 2);
if (receivedCRC != expectedCRC) {
errorMessage = "Received: ";
for (size_t i = 0; i < responseLen; i++) {
errorMessage += "[" + String(responsePacket[i], 16) + "]";
}
errorMessage += ", expected CRC: " + String(expectedCRC, 16);
return 0xFFFA;
}
value = (responsePacket[3] << 8) + responsePacket[4];
return 0;
}
void loop() {
/*
while(rs485Uart0->available()){
rs485Uart0->read();
}
while(rs485Uart1->available()){
rs485Uart1->read();
}
*/
String command = "smode stop";
xmitEna(0, true);
rs485Uart0->print(command + "\r");
rs485Uart0->flush();
xmitEna(0, false);
xmitEna(1, true);
rs485Uart1->print(command + "\r");
rs485Uart1->flush();
xmitEna(1, false);
delay(100);
String response0;
Serial.println("\r\n\n0 Response:");
for (int i=0; i<100; i++){
while(rs485Uart0->available()){
char c = rs485Uart0->read();
if(c != 0){
response0 += c;
}
}
delay(10);
}
Serial.println(response0);
Serial.println("\r\n\n1 Response:");
String response1 = "";
for (int i=0; i<100; i++){
while(rs485Uart1->available()){
char c = rs485Uart1->read();
if(c != 0){
response1 += c;
}
}
delay(10);
}
Serial.println(response1);
/*
uint16_t value;
String errorMessage;
int ret = holdingRegisterRead(rs485Uart0, 0, 0x0101, value,
errorMessage,
1000, 1000);
Serial.printf("RET0 %i, %i\n", ret, value * 10);
Serial.println(errorMessage);
*/
// Flush input buffer
}
If anyone can share any insight or point out all the wrong assumptions I'm making that would be super helpful. Also I was wondering if it might be possible to trigger the button press from the Raspberry Pi somehow...or if there's a remote way to trigger the LED Blink Restore? Could the buttons contacts be wired to a GPIO pin or something?? That'd be the ultimate fail-safe.
Last edited: