Teensys Intermittently Stop Responding?

Status
Not open for further replies.
I have a project that involves multiple (6) Teensys with audio boards connected via USB to a single computer running Ubuntu. Each Teensy is individually controlled by a Python process, and is set up to stream audio from the Python process (i.e., Teensy acts like a sound card). The Python scripts are written to run for an extended period of time (ideally days to weeks at a time). Teensys are in Serial + MIDI + Audio mode. Audio is being played at a sampling rate of 44.1 kHz.

Anywhere between an hour and a day or two after starting a Python script, one of the Teensys will stop receiving sensor input, or at least the attached sensors no longer trigger any reaction from the control program. It's not immediately noticeable because the attached sensors get triggered at irregular intervals that can range from a few seconds to hours. I don't know if that means the Teensy is not sending the information to the computer, or if the computer has stopped accepting the input for some reason. If I stop the Python process and restart it, the Teensy comes back up right away and the problem is resolved. Python doesn't report any errors, though, so I don't have much additional information.

I've used usbtop to see if there's still communication with the Teensys, and it looks like the computer is reporting communication in both directions, with an average PC->Teensy rate of 300 kb/s and an average Teensy->PC rate of 130 kb/s. I noticed that the Teensys that aren't responding seem to have an output rate of only 110 kb/s.

I have some theories, but not sure about the best way to proceed:
  • Since it resolves itself when I restart the Python process, is that a good indication that it's a problem on the Python side, or is there a way this could happen on the Teensy end?
  • Another possibility is that it's related to the USB bandwidth issues I reported in an earlier thread, but there's no sound distortion since I got a new USB hub.
  • Maybe the serial connection is crapping out for some reason after a length of time? I don't know if that would explain why there is still data coming from the Teensy... Maybe somehow the serial connection is getting intercepted by another program or something?

Does this kind of issue sound familiar for anyone? At the moment my solution is just to restart each Python script every day, but I'd like to come up with a more permanent solution.

I would post the Python code but it spans several dozen files...
For reference, here's the code uploaded to each Teensy:
Code:
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>
/*
 * Revision 1.1
 * 2/23/18 by AR
 * 
 */

/*
Pins used:
Set by Teensyduino:
//6 - Audio shield, MEMCS (NOT USED)
7 - Audio shield, MOSI (SD, shared)
9 - Audio shield, BCLK*
10 - Audio shield, CS (SD)
11 - Audio shield, MCLK*
12 - Audio shield, MISO (SD, shared)
13 - Audio shield, RX*
14 - Audio shield, SCLK (SD)
18 - Audio shield, SDA*
19 - Audio shield, SCL*
22 - Audio shield, TX*
23 - Audio shield, LRCLK*

Set by control program:
1 - trial light
2 - overhead light relay
3 - response light
16 - reinforcement relay
37 - response sensor
38 - start sensor

*/

// -----

static uint8_t myID[8]; //for getting unique Teensy ID
unsigned long serialNum = 0; //for getting Teensy serial number - not sure which to use, serialNum or myID


int inputValue = 0;
int baudRate = 19200;
char ioBytes[2];
int ioPort = 0;

// Audio setup
// GUItool: begin automatically generated code
AudioInputUSB            usb1;           //xy=200,69  (must set Tools > USB Type to Audio)
AudioOutputI2S           i2s1;           //xy=365,94
AudioConnection          patchCord1(usb1, 0, i2s1, 0);
AudioConnection          patchCord2(usb1, 1, i2s1, 1);
AudioControlSGTL5000     sgtl5000_1;     //xy=302,184
// GUItool: end automatically generated code


/*
Code for getting Teensy unique ID # (so log files can accurately reflect which unit they came from)
*/
void read_EE(uint8_t word, uint8_t *buf, uint8_t offset)  {
  noInterrupts();
  FTFL_FCCOB0 = 0x41;             // Selects the READONCE command
  FTFL_FCCOB1 = word;             // read the given word of read once area

  // launch command and wait until complete
  FTFL_FSTAT = FTFL_FSTAT_CCIF;
  while(!(FTFL_FSTAT & FTFL_FSTAT_CCIF))
    ;
  *(buf+offset+0) = FTFL_FCCOB4;
  *(buf+offset+1) = FTFL_FCCOB5;       
  *(buf+offset+2) = FTFL_FCCOB6;       
  *(buf+offset+3) = FTFL_FCCOB7;       
  interrupts();
}

    
void read_myID() {
  read_EE(0xe,myID,0); // should be 04 E9 E5 xx, this being PJRC's registered OUI
  read_EE(0xf,myID,4); // xx xx xx xx
  
  unsigned int ii;
  for (ii = 0; ii < sizeof(myID); ii++) {
    if ( ii > 3)
      serialNum = (serialNum << 8) + myID[ii];
  }
}


void setup(){
  Serial.begin(baudRate);
  while (!Serial) {
    ; // wait for serial port to connect
  }
  Serial.printf("Connected to Teensy");
  
  read_myID();
  Serial.printf("Teensy ID %d0 \n \n", serialNum);
  
  AudioMemory(12);
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.4);
  
}

void loop() {
  // All serial communications should be two bytes long
  // The first byte specifies the port to act on
  // The second byte specifies the action to take
  // The actions are:
  // 0: Read the specified input
  // 1: Write the specified output to HIGH
  // 2: Write the specified output to LOW
  // 3: Set the specified pin to OUTPUT
  // 4: Set the specified pin to INPUT
  // 5: Set the specified pin to INPUT_PULLUP
  // 6: Return Teensy ID number (pin independent)
  // 99: Audio control (pin independent)
  // if we get a valid serial message, read the request:
  if (Serial.available() >= 2) {
    // get incoming three bytes:
    Serial.readBytes(ioBytes, 2);
//    Serial.println("I received: ");
//    Serial.println(ioBytes[0], DEC);
//    Serial.println(ioBytes[1], DEC);
    // Extract the specified port
    ioPort = (int) ioBytes[0];
//  if (ioPort == 99) {
//    // 99 is for audio functions
//    if (ioBytes[1] == 0) {
//      //Stop playback
//      playSdRaw1.stop();  
//      break;
//    }
//    else if (ioBytes[1] == 1) {
//        //Selected type is S+, choose file
//        //randNumber = random(1,length(s+List))
//        //const char *filename = S+filelist[filenumber];
//    }
//    else if (ioBytes[1] == 2) {
//        //Selected type is S-, choose file
//        //randNumber = random(1,length(s-List))
//        //const char *filename = S-filelist[filenumber];
//    }   
//    //Could add additional types here
//    
//    playSdRaw1.play(filename);
//    break;
//  }
//
//  else
    // Specific pin actions 
    
    // Switch case on the specified action
    switch ((int) ioBytes[1]) {
      case 0: // Read an input
      inputValue = digitalRead(ioPort);
      Serial.write(inputValue);
      break;
      case 1: // Write an output to HIGH
      digitalWrite(ioPort, HIGH);       
      break;
      case 2: // Write an output to LOW
      digitalWrite(ioPort, LOW);        
      break;
      case 3: // Set a pin to OUTPUT
      pinMode(ioPort, OUTPUT);
      digitalWrite(ioPort, LOW);
      break;
      case 4: // Set a pin to INPUT
      pinMode(ioPort, INPUT);
      break;
      case 5: // Set a pin to INPUT_PULLUP
      pinMode(ioPort, INPUT_PULLUP);
      break;
      case 6: // Return Teensy ID
      Serial.write("Teensy ID: ");
      Serial.write(serialNum);
      break;
    //}
    }
  }
}
 
Unfortunately, the situation is about the same; between the six Teensy units, the disconnect happens at least once per day.

Quick Recap: I have six Teensys connected via USB to a PC running Ubuntu 16.04, and communicating through USB serial connection. The Teensys have identical programs (aside from a single character modification for unit ID): they only perform an action (either turning pins on/off or report pin state) when they receive a request from the computer. The serial connection is still failing regularly between a few hours after starting to a few days. For the most part, stopping and starting the Python script works and the serial connection is fine, until it fails again after a few hours.

There are two types of disconnects that seem to be occurring, from watching the USB communication speeds via usbtop.
  1. Communication speed drops: "To device" goes from ~300 kb/s to 275 kb/s, and "From device" drops from 130 kb/s to 107 kb/s.
    • I can replicate these readings if I open the serial monitor in a different program (like the Arduino IDE or tycommander) and connect it to the Teensy in question, so it seems like the serial connection to the python program is being interrupted. However, I'm not sure what program would be doing that.
  2. Less commonly, the communication speed drops out entirely (e.g. 0 kb/s in and out)
    • I have no idea why this one happens.
I can't seem to catch either scenario in the code consistently - I've added some steps to the Python script to check for a SerialException at every point (I think) where it communicates with the Teensy, and attempt to reestablish the connection, but it usually doesn't even detect the exception.
I have tried:
  • Getting a new external powered USB hub (that is probably Multi-TT)
  • Switching from a USB hub to a PCI USB card with dedicated controller
  • Moving the PC and power supplies to filtered power
The filtered power does seem to help some, in that the failure rate seems lower, but I still get those disconnects.
I could try to just build in a really robust connection checker and reconnector, but I'd really like to figure out what's causing the problem in the first place. I'm at a loss to explain it. My search results have not turned up any suitable explanations, other than the possibility of power fluctuations screwing with the connection, but I'm still having the issue on a filtered power source.

Here is the Teensy-specific python code:
Code:
import time
import datetime
import serial
import logging
from pyoperant.interfaces import base_
from pyoperant import utils, InterfaceError

logger = logging.getLogger(__name__)

class ArduinoInterface(base_.BaseInterface):
    """Creates a pyserial interface to communicate with an Arduino via the serial connection.
    Communication is through two byte messages where the first byte specifies the channel and the second byte specifies
    the action.
    Valid actions are:
    0. Read input value
    1. Set output to ON
    2. Set output to OFF
    3. Sets channel as an output
    4. Sets channel as an input
    5. Sets channel as an input with a pullup resistor (basically inverts the input values)
    :param device_name: The address of the device on the local system (e.g. /dev/tty.usbserial)
    :param baud_rate: The baud (bits/second) rate for serial communication. If this is changed, then it also needs to be
            changed in the arduino project code.
    """

    _default_state = dict(invert=False,
                          held=False,
                          )

    def __init__(self, device_name, baud_rate=19200, inputs=None, outputs=None, *args, **kwargs):

        super(ArduinoInterface, self).__init__(*args, **kwargs)

        self.device_name = device_name
        self.baud_rate = baud_rate
        self.device = None

        self.read_params = ('channel', 'pullup')
        self._state = dict()
        self.inputs = []
        self.outputs = []

        self.open()
        if inputs is not None:
            for input_ in inputs:
                self._config_read(*input_)
        if outputs is not None:
            for output in outputs:
                self._config_write(output)

    def __str__(self):

        return "Arduino device at %s: %d input channels and %d output channels configured" % (
            self.device_name, len(self.inputs), len(self.outputs))

    def __repr__(self):
        # Add inputs and outputs to this
        return "ArduinoInterface(%s, baud_rate=%d)" % (self.device_name, self.baud_rate)

    def open(self):
        """Open a serial connection for the device
        :return: None
        """

        logger.debug("Opening device %s" % self)
        self.device = serial.Serial(port=self.device_name,
                                    baudrate=self.baud_rate,
                                    timeout=5)
        if self.device is None:
            raise InterfaceError('Could not open serial device %s' % self.device_name)

        logger.debug("Waiting for device to open")
        self.device.readline()
        self.device.flushInput()
        logger.info("Successfully opened device %s" % self)

    def close(self):
        """Close a serial connection for the device
        :return: None
        """

        logger.debug("Closing %s" % self)
        self.device.close()

    def _config_read(self, channel, pullup=False, **kwargs):
        """ Configure the channel to act as an input
        :param channel: the channel number to configure
        :param pullup: the channel should be configured in pullup mode. On the arduino this has the effect of
        returning HIGH when unpressed and LOW when pressed. The returned value will have to be inverted.
        :return: None
        """

        logger.debug("Configuring %s, channel %d as input" % (self.device_name, channel))
        if pullup is False:
            self.device.write(self._make_arg(channel, 4))
        else:
            self.device.write(self._make_arg(channel, 5))

        if channel in self.outputs:
            self.outputs.remove(channel)
        if channel not in self.inputs:
            self.inputs.append(channel)

        self._state.setdefault(channel, self._default_state.copy())
        self._state[channel]["invert"] = pullup

    def _config_write(self, channel, **kwargs):
        """ Configure the channel to act as an output
        :param channel: the channel number to configure
        :return: None
        """

        logger.debug("Configuring %s, channel %d as output" % (self.device_name, channel))
        self.device.write(self._make_arg(channel, 3))
        if channel in self.inputs:
            self.inputs.remove(channel)
        if channel not in self.outputs:
            self.outputs.append(channel)
        self._state.setdefault(channel, self._default_state.copy())

    def _read_bool(self, channel, **kwargs):
        """ Read a value from the specified channel
        :param channel: the channel from which to read
        :return: value

        Raises
        ------
        ArduinoException
            Reading from the device failed.
        """

        if channel not in self._state:
            raise InterfaceError("Channel %d is not configured on device %s" % (channel, self.device_name))

        if self.device.inWaiting() > 0:  # There is currently data in the input buffer
            self.device.flushInput()
        self.device.write(self._make_arg(channel, 0))
        # Also need to make sure self.device.read() returns something that ord can work with. Possibly except TypeError
        while True:
            try:
                v = ord(self.device.read())
                break
            except serial.SerialException:
                # This is to make it robust in case it accidentally disconnects or you try to access the arduino in
                # multiple ways
                raise ArduinoException("Serial connection interrupted")
            except TypeError:
                ArduinoException("Could not read from arduino device")

        logger.debug("Read value of %d from channel %d on %s" % (v, channel, self))
        if v in [0, 1]:
            if self._state[channel]["invert"]:
                v = 1 - v
            return v == 1
        else:
            logger.error("Device %s returned unexpected value of %d on reading channel %d" % (self, v, channel))
            # raise InterfaceError('Could not read from serial device "%s", channel %d' % (self.device, channel))

    def _poll(self, channel, timeout=None, wait=None, suppress_longpress=True, **kwargs):
        """ runs a loop, querying for presses. returns trigger time or None if polling times out
        :param channel: the channel from which to read
        :param timeout: the time, in seconds, until polling times out. Defaults to no timeout.
        :param wait: the time, in seconds, between subsequent reads. Defaults to 0.
        :param suppress_longpress: only return a successful read if the previous read was False. This can be helpful
        when using a button, where a press might trigger multiple times.

        :return: timestamp of True read
        """

        if timeout is not None:
            start = time.time()

        logger.debug("Begin polling from device %s" % self.device_name)
        while True:
            if not self._read_bool(channel):
                logger.debug("Polling: %s" % False)
                # Read returned False. If the channel was previously "held" then that flag is removed
                if self._state[channel]["held"]:
                    self._state[channel]["held"] = False
            else:
                logger.debug("Polling: %s" % True)
                # As long as the channel is not currently held, or longpresses are not being supressed,
                # register the press
                if (not self._state[channel]["held"]) or (not suppress_longpress):
                    break

            if timeout is not None:
                if time.time() - start >= timeout:  # Return GoodNite exception?
                    logger.debug("Polling timed out. Returning")
                    return None

            # Wait for a specified amount of time before continuing on with the next loop
            if wait is not None:
                utils.wait(wait)

        self._state[channel]["held"] = True
        logger.debug("Input detected. Returning")
        return datetime.datetime.now()

    def _write_bool(self, channel, value, **kwargs):
        """Write a value to the specified channel
        :param channel: the channel to write to
        :param value: the value to write
        :return: value written if succeeded
        """

        if channel not in self._state:
            raise InterfaceError("Channel %d is not configured on device %s" % (channel, self))

        logger.debug("Writing %s to device %s, channel %d" % (value, self, channel))
        if value:
            s = self.device.write(self._make_arg(channel, 1))
        else:
            s = self.device.write(self._make_arg(channel, 2))
        if s:
            return value
        else:
            raise InterfaceError('Could not write to serial device %s, channel %d' % (self.device, channel))

    @staticmethod
    def _make_arg(channel, value):
        """ Turns a channel and boolean value into a 2 byte hex string to be fed to the arduino
        :return: 2-byte hex string for input to arduino
        """

        return "".join([chr(channel), chr(value)])


class ArduinoException(Exception):
    pass

There's another issue that might be relevant: when I was using the external USB hub, after awhile one of the ports on the hub stopped responding and didn't report anything connected at all. If I plugged the Teensy from that port directly into the computer, it would appear as a device, but it acted like the port on the hub was dead. Then, in the middle of troubleshooting, the whole hub 'died' and my computer reported that no Teensys were connected at all, even though I could plug each in directly and it would be fine. That was resolved by plugging the hub into a different port on the back (and all ports on the hub started working again), but I don't know what caused it in the first place.
 
Status
Not open for further replies.
Back
Top