USB Serial Sensitive to NULL? Can't be.

Status
Not open for further replies.
This can't be, yet I've run into a problem I can't track down. I'm trying to send binary data using the USB HID serial support. There's no problem with data sent from the Teensy to the host. All is perfect. 1 to 64 byte chunks of binary data. Life is good.

I'm using the code below to attempt to receive binary data from the host. Until now, I've been developing and testing by sending just one byte, which is never zero, and had no trouble. Now that I'm trying to send chunks of 6 bytes, the code below fails as follows:

If I send 0x01 0x02 0x03 0x04 0x05 0x06 it arrives fine and the byte count is correct.

If I send 0x01 0x02 0x00 0x04 0x05 0x06 the byte count is two and only the first two bytes are received. Experimenting, I found that it's acting like NULL is a terminator. I can send any number of non-zero bytes but as soon as it hits a data byte of 0x00 it stops receiving.

At the sending end (Linux box) I'm using hidapi.h and the hid_write(buf,count) function, which works by byte count, not the content of the data.

I'm sure I'm missing something dumb but I can't see what it is.

Code:
char usbBuf[65];

// Get latest packet of USB data.
int getUSB() {
  int count = 0;
  while(Serial.available() && count < 64) {
    usbBuf[count++] = Serial.read();
    }
  return(count);
}
 
Indeed - not seeing the code it could be some number of things. count=strlen(buffer) >> write( buffer, count)?

re: "sending just one byte, which is never zero, and had no trouble" - was this one byte version tested using zero?
 
Here's the whole program as of now:

Code:
// Teensy 3.2
// USBtoLED.c

#include <OctoWS2811.h>

// Machine instructions to restart the processor
// Execute with CPU_RESTART
// Not needed in this application
#define CPU_RESTART_ADDR (uint32_t *)0xE000ED0C
#define CPU_RESTART_VAL 0x5FA0004
#define CPU_RESTART (*CPU_RESTART_ADDR = CPU_RESTART_VAL);

// Program declarations
char usbBuf[65];
int tik, interval30secs;
boolean ledOn;
unsigned long secInterval;

// LED Library Declarations
const int ledsPerStrip = 50;  // (LEDs per channel)
DMAMEM int displayMemory[ledsPerStrip*6];
int drawingMemory[ledsPerStrip*6];
const int config = WS2811_GRB | WS2811_800kHz;

OctoWS2811 leds(ledsPerStrip, displayMemory, drawingMemory, config);



// Once
void setup() {

  pinMode(13, OUTPUT); //set LED pin as output

  Serial.begin(9600); //USB is always 12 Mbit/sec

  tik = 0;
  interval30secs = 2;  //startup at just 2 seconds, then 30
  ledOn = false;
  secInterval = millis() + 1000;

  leds.begin();
  leds.show();
}

// Main loop -- loops forever

void loop() {
  int count;

//First we run the schedulers
  if( millis() > secInterval ) {
    //these tasks run once per second
    tik++;

    if( ledOn ) {
      digitalWrite(13, LOW);    // set the LED off
      ledOn = false;
      }
    else {
      digitalWrite(13, HIGH);   // set the LED on
      ledOn = true;
      }

    if( tik > interval30secs ) {
      //these tasks execute every 30 seconds
      interval30secs = tik+30;
      //send heartbeat
      Serial.print("HB ");
      Serial.print(tik,DEC);
      Serial.send_now();
      }

    secInterval += 1000;
    }


  if( (count = getUSB()) ) {
    processCmd(count);
    }

}

// Get latest packet of USB data.
int getUSB() {
  int count = 0;
  while(Serial.available() && count < 64) {
    usbBuf[count++] = Serial.read();
    }
  return(count);
}


// Process command. Returns "##" as handshake to acknowledge that
// command(s) are processed and another packet can be sent.
// ## does not indicate that there were no errors, just that
// the packet was received and processed.  "##" is also the heartbeat.
//
// USB packets consist of 64 bytes. The first byte specifies the command,
// 0 to 255. The remaining 63 bytes are utilized (or not) depending on
// the command.
//
// Command 0 is not used.
//
// Command 1 is the heartbeat command. Single byte command.
//
// Command 2 sets the color in the memory buffer for up to 15 LEDs, but
//  does not trigger an update of the LEDs. (see Command 3)
//  Byte 0: 2
//  Byte 1: Channel number, 0 to 7.
//  Byte 2: Number of LED elements in this command, 1 to 15.
//    An element consists of four bytes. The first byte is the LED number,
//    0 to 255. Then a three byte RGB, GBR, BRG, etc. color triplet.
//  Byte 3: not used
//  Bytes 4 to 63: as above
//
// Command 3 triggers an update of all LEDs. Single byte command.

boolean processCmd(int count) {
  int cmdNum;
  int channelNum;
  int numElems;
  int ledNum;
  unsigned int color;
  int i;
  char *cp;

  cmdNum = usbBuf[0];
  switch( cmdNum ) {
    case 0:  //not used
      return(false);
    case 1:  //heartbeat (send cmd ack only)
      break;
    case 2:
      channelNum = usbBuf[1];
      numElems = usbBuf[2];
      if( channelNum < 0 || channelNum > 7 ) return(false); //error
      if( numElems < 1 || numElems > 15 ) return(false); //error
      cp = usbBuf+4;

      for( i=0; i<numElems; i++ ) {
        ledNum = (int)cp[0];
        //The following code must be adjusted for RGB, GRB, BRG
        color = (unsigned int)cp[1] << 16;
        color |= (unsigned int)cp[2] << 8;
        color |= (unsigned int)cp[3];
        leds.setPixel((channelNum * ledsPerStrip) + ledNum, color);
        cp += 4;
        }
      break;
    case 3:
      leds.show();
      break;
    default:
      return(false);  //error
    }

//$$$$$ debugging code added to cmd ack
  Serial.print("##cnt=");  //send cmd ack
  Serial.print(count,DEC);
  Serial.print(">>");
  Serial.write(usbBuf,count);
  Serial.send_now();
  return(true);
}
 
That is the Teensy side that can send but not get complete receive - showing a set of notTheTeensy send side code that is enough to see the issue might yield something.
 
Sounds like the code you're seeing above shouldn't / can't be sensitive to data that contains embedded zero values. If that's the case, the problem must be at the sending end, at the host. The code at that end is way too big to post here in its entirety. Talking over USB is just a tiny fraction of what it does. There must be something funny going on in the hidapi.h stuff.

I'll see if I can dig deeper there.
 
Quick run through the code seemed orderly - as long as a USB packet it filled with the whole message - and the whole message is received.

Perhaps write a byte of 0xff before and after the message 'buf' outside the current write - something to confirm the send is working end to end in each packet.
 
Good idea. And, thank you.

I half expected someone to say, "Oh yeah. USB serial has binary and text modes. You have to set a switch #define SERIAL_BINARY (1)." Or, something to that effect. Apparently, that's not the case. Lol. (I'm a newbie at this Arduino stuff.)
 
Hmm. Thanks for the link. I'll study that.

But, isn't the Teensy different, using its own versions of serial and everything else? I've been taking Arduino documentation with a grain of salt because the Teensy isn't an Arduino. It's similar, but a different species.
 
This is on the Teensy side. Regular USB Serial supports binary data, but the USB HID emulation of serial does not.

For full binary data on HID, you need to use RawHID.
 
If anything 'Arduino' doesn't work the same ( or better ) - with FEW exceptions - you need to let Paul know so he can correct Teensy behavior.

Not sure there are any high level exceptions as the goal is Arduino compatibility. The only issues come up in low level code to produce the results (typically on device libraries) that are ARM processor specific and not always compatible with the AVR/UNO type way of doing things.
 
To explain a bit further, since this probably isn't documented anywhere, when you select anything in Tools > USB Type which does *not* include "Serial", a HID interface is used to emulate serial. It's meant so Arduino programs which use Serial.print() and Serial.read() can still talk to the Arduino Serial Monitor.

Unfortunately, HID is not a byte-based protocol like Serial. With regular Serial, when your PC wants to send 7 bytes to Teensy, one USB packet with exactly 7 data bytes is sent. Or the PC could send one 3 byte packet and one 4 byte packet, or seven single byte packets. With HID, data is sent in "reports" which are fixed size. This is simply how HID protocol works. It's designed for stuff like keyboards or mice, where each packet is an update with the keystrokes or mouse movement & button state, which is always a fixed size. So to emulate serial, those 7 bytes would be put into the beginning of a 32 byte packet. The other 25 required bytes are padded with zeros. On the receiving end, zeros in the data are interpreted as padding and discarded.

This serial emulation was never meant to be a general purpose data communication channel. It was meant for allowing existing Arduino programs to talk to the Arduino Serial Monitor. Since the serial monitor doesn't display 0 bytes, and never transmits them, normally this isn't an issue. We don't publish info about using the serial emulation HID interface. But if you look at the code or what your PC detects from the descriptors and try to use it, this not-so-documented behavior where zeros are considered padding can come up. As far as I can recall, you may be the very first person to ever get this far and hit this limitation.

Since 2009, Teensy has offered "RawHID" as the documented & supported way to do generic HID protocol communication. Instead of giving you bytes, you send and receive full packets. Full binary data is supported. There no hidden parsing code trying to emulate Serial, deciding which bytes were user data and which were necessary padding to reach the HID report size. We call it RawHID because you get access to the raw 64 byte packets, without any HID parser or any other code doing stuff to the data.

To get started with RawHID, in Arduno click File > Examples > Teensy > USB_RawHID > Basic. You'll also need to set Tools > USB Type to RawHID. When you communicate with Teensy, you'll see 2 HID interfaces. One is the serial emulation, which you shouldn't use, except from the serial monitor - which is really handy since you can use Serial.print() independently. The other is the RawHID. That's the interface you should use with the hidapi library.

You might also keep in mind the HID bandwidth limitation, if you're going to expand this project to a larger number of LED and try to achieve 30 or 60 Hz refresh rates. With HID, you're limited to at most 1000 reports per second. So you just can't go faster than 64,000 bytes per second. However, if other bandwidth hogging devices are active, HID does guarantee you'll be able to get this speed.

With regular USB serial, you can use all the available USB bandwidth. But "available" means all the bandwidth that's not currently consumed by other devices. Usually you get quite a lot of bandwidth, but there's no guarantee how much will be left over if other bandwidth hogging devices are consuming a lot of USB bandwidth.
 
Hello Paul. Thank you very much for your comprehensive answer.

Sorry for the delay in responding. Real life interfered. Lol.

I should have mentioned that RawHID has been my only focus. The first thing I did when I first lit up the Arduino development system was set Tools / USB Type to HID and it's been there ever since.

One mistake I made was not knowing about the RawHID function calls. With the mode set to RawHID in the development system, I figured (wrongly) that I should still use the Arduino style Serial calls -- and they mostly worked. But 64 byte binary buffers up and down is all I want. Bandwidth is not an issue for anything I foresee and transactions are capped at 10 per second at the host. Reality is more like 2 or 3 per minute.

Background: I've used USB for many years but never really learned it. Projects were always on a deadline and when I looked at USB documentation, I didn't have time to learn something that complex. So I cheated and used an FTDI chip.

This project is not a rush, so I decided to finally learn USB. I'm no expert, but now I understand the basics of how it works and why it was designed as it is. I settled on hidapi.h as a decent solution and built a library that does the things I want. I enumerate all USB devices and put that in a table. Then I search for what I want. I discovered the Teensy enumerates as two devices as follows:

Device Found
type: 16c0 0486
path: 0002:0006:00
serial_number: 5527490
Manufacturer: Teensyduino
Product: Teensyduino RawHID
Release: 275
Interface: 0

Device Found
type: 16c0 0486
path: 0002:0006:01
serial_number: 5527490
Manufacturer: Teensyduino
Product: Teensyduino RawHID
Release: 275
Interface: 1

At first this puzzled me, but then I realized how it works. Connecting by means of Manufacturer/Product ID always connects to Device 0 and, as you point out, didn't work. To connect to Device 1, I had to change to using the full USB path. That worked.

However, I was in raw HID mode at both ends of the link but was using the Serial calls in the Teensy. This worked until I ran into the NULL problem.

Now, I've modified the Teensy code to use RawHID.send, etc., which was easy because what I wanted all along is 64 byte binary packets and I'll worry about what's in the packet.

I expected it to work straight off, but it doesn't. Haha. I'm getting nothing at present.

I'm confused on a couple of things:

The RawHID example in the Arduino dev sys uses RawHID.send() and RawHID.recv() and no USB initialization calls. That's what I went with.

However, the documentation on the web site talks about different functions like:

void usb_init(void);
uint8_t usb_configured(void);
int8_t usb_rawhid_recv(uint8_t *buffer, uint8_t timeout);

Also, I assume that since I was able to establish communications before simply by connecting to the USB full path, parameters like these:

#define RAWHID_USAGE_PAGE 0xFFAB // recommended: 0xFF00 to 0xFFFF
#define RAWHID_USAGE 0x0200 // recommended: 0x0100 to 0xFFFF

...are not necessary. In fact, I don't even know what I would do with those. hidapi.h doesn't require them.

So I enumerate USB devices. The Teensy reports two devices, 0 and 1. I connect with hid_open_path(path), which returns a handle and no errors. But then I can't write to it. hid_write() returns -1 (error). And it didn't give errors before, when I was using the Serial calls in the Teensy. So I'm wondering if there's some initialization step at the Teensy that I need to be doing. Is it possible that the first call to a Serial function like Serial.println() triggers some additional needed initialization?

The code now looks like this:

Code:
// Teensy 3.2
// USBtoLED.c

#include <OctoWS2811.h>

// Machine instructions to restart the processor
// Execute with CPU_RESTART
// Not needed in this application
#define CPU_RESTART_ADDR (uint32_t *)0xE000ED0C
#define CPU_RESTART_VAL 0x5FA0004
#define CPU_RESTART (*CPU_RESTART_ADDR = CPU_RESTART_VAL);

// Program declarations
byte usbBuf[65];
int tik, interval30secs;
boolean ledOn;
unsigned long secInterval;

// LED Library Declarations
const int ledsPerStrip = 50;  // (LEDs per channel)
DMAMEM int displayMemory[ledsPerStrip*6];
int drawingMemory[ledsPerStrip*6];
const int config = WS2811_GRB | WS2811_800kHz;

OctoWS2811 leds(ledsPerStrip, displayMemory, drawingMemory, config);



// Once
void setup() {

  pinMode(13, OUTPUT); //set LED pin as output

  Serial.begin(9600); //USB is always 12 Mbit/sec

  tik = 0;
  interval30secs = 2;  //startup at just 2 seconds, then 30
  ledOn = false;
  secInterval = millis() + 1000;

  leds.begin();
  leds.show();
}

// Main loop -- loops forever

void loop() {
  int count;

//First we run the schedulers
  if( millis() > secInterval ) {
    //these tasks run once per second
    tik++;

    if( ledOn ) {
      digitalWrite(13, LOW);    // set the LED off
      ledOn = false;
      }
    else {
      digitalWrite(13, HIGH);   // set the LED on
      ledOn = true;
      }

    if( tik > interval30secs ) {
      //these tasks execute every 30 seconds
      interval30secs = tik+30;
      //send heartbeat
      usbBuf[0] = 'H';
      usbBuf[1] = 'B';
      RawHID.send(usbBuf,70);
      }

    secInterval += 1000;
    }


  if( (count = RawHID.recv(usbBuf,0)) ) {
    processCmd(count);
    }

}

// Process command. Returns "##" as handshake to acknowledge that
// command(s) are processed and another packet can be sent.
// ## does not indicate that there were no errors, just that
// the packet was received and processed.  "##" is also the heartbeat.
//
// USB packets consist of 64 bytes. The first byte specifies the command,
// 0 to 255. The remaining 63 bytes are utilized (or not) depending on
// the command.
//
// Command 0 is not used.
//
// Command 1 is the heartbeat command. Single byte command.
//
// Command 2 sets the color in the memory buffer for up to 15 LEDs, but
//  does not trigger an update of the LEDs. (see Command 3)
//  Byte 0: 2
//  Byte 1: Channel number, 0 to 7.
//  Byte 2: Number of LED elements in this command, 1 to 15.
//    An element consists of four bytes. The first byte is the LED number,
//    0 to 255. Then a three byte RGB, GBR, BRG, etc. color triplet.
//  Byte 3: not used
//  Bytes 4 to 63: as above
//
// Command 3 triggers an update of all LEDs. Single byte command.

boolean processCmd(int count) {
  int cmdNum;
  int channelNum;
  int numElems;
  int ledNum;
  unsigned int color;
  int i;
  byte *cp;

  cmdNum = usbBuf[0];
  switch( cmdNum ) {
    case 0:  //not used
      return(false);
    case 1:  //ping command (send cmd ack only)
      break;
    case 2:
      channelNum = usbBuf[1];
      numElems = usbBuf[2];
      if( channelNum < 0 || channelNum > 7 ) return(false); //error
      if( numElems < 1 || numElems > 15 ) return(false); //error
      cp = usbBuf+4;

      for( i=0; i<numElems; i++ ) {
        ledNum = (int)cp[0];
        //The following code must be adjusted for RGB, GRB, BRG
        color = (unsigned int)cp[1] << 16;
        color |= (unsigned int)cp[2] << 8;
        color |= (unsigned int)cp[3];
        leds.setPixel((channelNum * ledsPerStrip) + ledNum, color);
        cp += 4;
        }
      break;
    case 3:
      leds.show();
      break;
    default:
      return(false);  //error
    }

  // echo most of the received buffer contents (first 50 bytes)
  // back to the host for examination.
  // shift buffer contents up a ways to make room for cmd ack
  memcpy(usbBuf,usbBuf+7,50);
  memcpy(usbBuf,"##cnt",5);
  usbBuf[5] = lowByte(count);
  usbBuf[6] = '#';
  i = RawHID.send(usbBuf,70);
  if( i ) return(true);
  return(false);
}
 
Trying different things here, I noted more confusion on my part. Sorry.

I fired up the Basic Raw HID Example program, as is. It runs and produces the text output messages on the Arduino terminal. It prints at 2 second intervals about ten lines, then switches to "Unable to transmit packet".

I also find that if I run my host program instead of the terminal, I get those same messages there, and the same thing happens as above.

So my host program is apparently connecting and able to receive from the Teensy. But sending from the host to the Teensy, which worked before, now doesn't. How odd. Lol.
 
Okay. Have to do more testing, but just for fun I tried connecting to Device 0 instead of Device 1 and Raw HID communications seem to be working. Looks like I have another bug or two hiding in there -- possibly errors in my debugging code. But I think I'm getting the picture now.

I thought the functions of Device 0 and Device 1 were the other way around.
 
Yes! Fixed! It actually worked first time as soon as I switched to using Device 0. I couldn't see it because there were two bugs, both in my patchwork debug code I'd stuck in there. Once I ripped that out, it all works as it's supposed to at this point in the development.

Thank you, Paul, for creating such insanely cool products and made to very high quality standards. :)
 
Glad you got it figured out.

Any chance you might have minimal working hidapi-based code to share? Something that works together with the "Basic" example would be perfect. Might really help anyone else struggling with hidapi, who later finds this thread by searching.
 
Yeah. I'll look into stripping it down to essentials and removing "clever" stuff I do in larger systems, like using associative arrays to hold data from config files. For example, the manufacturer and product codes for the Teensy are in the program's external config file.

I mentioned that during development I learned that Teensy Device 1 is where I thought I needed to connect so I avoided Device 0, which I assumed was for your loader, etc. So I had to connect using the full USB path and couldn't use the easier VID/PID method. Now, I know that for raw HID I need to connect to Device 0, which means you can use the simpler VID/PID method. So, perhaps I'll do two examples. One simple one that blindly connects to the first Teensy it finds via VID/PID and another that shows how to enumerate all HID devices and connects using full path. The second method is more complex but lets you identify and connect to any number of Teensies, or other raw HID devices that might be plugged in at once.

This is sounding like a job for Git. I've never used Git for anything. Hah. Yikes.

Thanks again for your cool products. Now that the misunderstandings are cleared up, it's working rock solid. Although, I still have more to learn.
 
When it comes to examples, the smallest & simplest code that "just works" when used together with some other specific example code is usually best. Usually just getting to something that works is the hard part, so the simplest thing that gets to working communication is best.

Git is awesome. I use it almost every day. It's also tremendously complex. I've never learned most of the stuff it can do. I just use a few of the simplest commands, and I have a "cheat sheet" file for a several more when I do stuff like contribute code to Arduino.

As far as sharing a simple hidapi example, even just posting the code in a message on this thread would really help. If you do, I'll add a link to the message in the RawHID Basic example. If someone else is struggling to get this working, I'm sure they'll be just as happy to copy and paste from a message here as they would getting it from a github repository. :)
 
That's good advice. I'll do that. Since it's Device 0 that we're after, a highly stripped down version that "just works" is possible. Fancy stuff can be shown in other examples.

I believe that a stripped down version is small enough to post here. I'll find out.
 
Status
Not open for further replies.
Back
Top