Serial splits string?

[AdmiralCrunch]

Hi,

when a switchbutton is pressed, I am sending a string via Serial1 from a Teensy3.2 to a Raspi3, where I catch it with NodeJS.

#include <Bounce.h>
#include <cfg.h>

cfg cfg;
Bounce btnShift = Bounce(12, 10);  // 10 ms debounce


void setup() {
  pinMode(12, INPUT_PULLUP);

  Serial.begin(9600);
  while ( !Serial && (millis() < 4000) ) {
    Serial.println("LOADING SERIAL PORT ...");
  }
  Serial1.begin(9600);
  while ( !Serial1 && (millis() < 4000) ) {
    Serial1.println("LOADING Serial1 PORT ...");
  }

  Serial.println("SERIAL PORT INITIALIZED");
  Serial.println("Serial1 PORT INITIALIZED");
}

void loop() {
  pollShift();

}


   void pollShift() {
    if (btnShift.update()) {
      if (btnShift.fallingEdge()) {
          cfg.shiftStatus = true;
          Serial.println("Shift pressed..");          
          Serial1.write("bla:110");
      } else {
        Serial.println("Shift released..");
          cfg.shiftStatus = false;
      }
    }
  }

Everything works fine but If I send a string greater then 8 chars, Serial1.send() seems to split the request into two.. oO .. what is that? how can I fix that?

 

[defragster]

Serial1 from Teensy shouldn't have any splitting behavior. Maybe if the buffer was overflowing or some break - but that will be at 64 bytes.

Have you run a common ground between the two?

It seems this is on the Raspi3 end you are seeing this? The Serial1 baud should work at or well above 115200 baud - perhaps see if faster transmit changes the observed behavior.

 

[Projectitis]

Not sure if this helps or not, but during your init code you may be filling the Serial buffer up with data. I'm not entirely sure how Serial works but this is my understanding:

Serial.begin(9600);

// Do this loop while the Serial is NOT ready and it's less than 4000 millis since program started
while ( !Serial && (millis() < 4000) ) {
	// Even though Serial is not ready, we are 'sending' this message via Serial. I don't know
	// what actually happens here - if this data is ignored, or if it is filling up the Serial
	// buffer? It is much better for there to be nothing in this loop, or perhaps a short delay.
	Serial.println("LOADING SERIAL PORT ...");
}

Serial1.begin(9600);

// Do this loop while the Serial1 is NOT ready and it's less than 4000 millis since program started
while ( !Serial1 && (millis() < 4000) ) {
	// Even though Serial1 is not ready, we are 'sending' this message via Serial1. I don't know
	// what actually happens here - if this data is ignored, or if it is filling up the Serial1
	// buffer? It is much better for there to be nothing in this loop, or perhaps a short delay.
	Serial1.println("LOADING Serial1 PORT ...");
}

This code would be better - but whether it solves your problem or not, I don't know:

uint32_t start;

start = millis();
Serial.begin(9600);
while ( !Serial && ((millis()-start) < 4000) ) {
	delay(1);
}

start = millis();
Serial1.begin(9600);
while ( !Serial1 && ((millis()-start) < 4000) ) {
	delay(1);
}

p.s. The changes to "millis()" above ensure that each serial channel has 4 seconds (4000 milliseconds) to initialize. The original code just checked for 4000 milliseconds since the counter started (i.e. when the teensy board was powered up). For example, this meant that if "Serial" took 3500 milliseconds to start up, "Serial1" would only be given 500 milliseconds, which might not be enough.

 

[defragster]

I wondered about that - there is nothing to hold up "Serial1.begin(9600);" like USB Serial - so that test will pass AFAIK and based on testing with this code where there is nothing printed to Serial with this in setup() ... Either from the IF or the WHILE.

  if ( !Serial1 ) {
    Serial.println("BEFORE Serial1.begin() Serial1 PORT ...");
  }
  Serial1.begin(9600);
  while ( !Serial1 && (millis() < 4000) ) {
    Serial.println("LOADING Serial1 PORT ...");
  }

 

[Projectitis]

So this startup code should be ok then:

uint32_t start  = millis();

Serial1.begin(9600);
Serial.begin(9600);
while ( !Serial && ((millis()-start) < 4000) ) {
	delay(1);
}

But whether it solves the 'splitting' problem - who knows!

 

[defragster]

Yes that should work - but not sure it was causing trouble before - though not useful. And the raspi3 would have dumped that data out.

Might be that the binary Serial1.write versus Serial1.print usage

 

[AdmiralCrunch]

Yes that should work - but not sure it was causing trouble before - though not useful. And the raspi3 would have dumped that data out.

Might be that the binary Serial1.write versus Serial1.print usage

On the Raspi-side I get the Buffer. I then do a buffer.toString('ascii'); to get the original string that was send.

What makes me curious is, that when I do Serial1.write("blaa:110");, I get the string on Raspi as it it.. in one piece.
But, when I do Serial1.write("blaaa:110");, then I get first request with "blaaa:11" and then comes a second request with "0"

I suggest on the RaspiSide is, everything is ok.. I don't do any magic there, yet.. on NodeJS I require and initialize "serialport" and then do sp.on('data', ());

I have read that Teensy-Serial is 8 bit.. and I am not able to send more then 8 chars at once.. maybe that has somethin to do with it?

 

[defragster]

Serial is an 8 bit protocol - by default. Each 'character' is counted/stored after 8 data bits are received. That would not relate to the issue seen with 8 characters.

What happens using : Serial1.write versus Serial1.print

Also the baud rate being slow may - not sure if the raspi3 is timing out - or catching between incoming characters and assuming the transmission is complete when read is requested? Without a terminating character [ newline, return or other special character ] what is in the buffer is based on what is received at the time it is requested.

The problem likely in how the chars are read in the raspi3 - and needs to be adjusted to account for what is expected to be sent - and how it is parsed.

 

[macrosii]

I have a similar App T3.5 to Rap3+ and also Ras2. I was using in the PI native UART at 9600 and experienced similar multiple calls to "port.on", after between 6 and 10 bytes. It did not seem to cause a problem as they were processed in order and the data ended up ok in a global buffer.
I am now using USB FTDI and the driver serves data to the Linux kernel after receiving a packet of less than 64 bytes or after 18ms. So at 9600 you would get 15 bytes.
The problem I can see is what happens on a subsequent call to port.on(event2) if it tries to populate the global buffer before port.on(event1) finishes. I believe in my code I need to add some sort on lock in the node code on the global buffer counter (to be safe)

 

[AdmiralCrunch]

I have a similar App T3.5 to Rap3+ and also Ras2. I was using in the PI native UART at 9600 and experienced similar multiple calls to "port.on", after between 6 and 10 bytes. It did not seem to cause a problem as they were processed in order and the data ended up ok in a global buffer.
I am now using USB FTDI and the driver serves data to the Linux kernel after receiving a packet of less than 64 bytes or after 18ms. So at 9600 you would get 15 bytes.
The problem I can see is what happens on a subsequent call to port.on(event2) if it tries to populate the global buffer before port.on(event1) finishes. I believe in my code I need to add some sort on lock in the node code on the global buffer counter (to be safe)

hi

that maybe could help. What do you mean by "global buffer", how can I access it?

 

[macrosii]

I am sending commands between the two, with start and end bytes, CRC, and packet length (max 64 bytes) so need to be able to assemble the received data into a buffer so that it can then be processed. So in node something like "var inDataByte = new Buffer(64);" in global scope so that it is persistent between successive port.on events.

 

[KurtE]

Am I missing something here. That is are you assuming there is something wrong here?

That is if you send lets 10 or 100 bytes over Serial1 that is to be read by the RP3, you are expecting that RPI3, will read all of it in one buffer? I would sure hope not!

That is as @Defragster mentioned, Serial1 is just a simple 8 bit Serial port, it just is going to stream bits out at 960 characters per second more or less(8 bits plus start plus stop bit) The bytes are simply streamed one after another...
The Teensy should never duplicate a character going out...

Might help to know how Serial1 is connected to your RPI3? Through USART on connector, some USB to serial adapter...

If some form of USB to serial adapter, very potentially the adapter is buffering so many characters at a time (or timeout) and then sending the buffer of characters as one packet over USB... Maybe 8 characters?


But regardless of this, if you have a simple Serial receiving code on the RPI3, which receives data. It might work similar to Arduino code written something like:

uint16_t ReadSerial(uint8_t *buffer, uint16_t cb) {
    uint16_t avail_to_read = Serial3.available();
    if (!avail_to_read) return 0;
    if (avail_to_read > cb) avail_to_read = cb;
    Serail3.read(buffer, avail_to_read);
    return avail_to_read;
}

This code assuming I typed it correctly, will simply check to see if there are any characters that have been transmitted and return them now... It again knows nothing about what the characters are.
So if you send a string, it may give you the whole string or it might give you partial. It may easily depend on the internals of your NodeJs. I don't use NodeJs or Java... So don't know their options. But there may be some option to tell it that you wish for it to process Text lines, that end in some terminator (CR or NL or combination). And then it may not return to your code until it receives a whole string?

But again maybe it would help to see complete description of setup? Like how the two are connected, Are you using some USB to serial adapter? Using UART? How was it configured. I assume you configured it to disable the console, ...
And what code is running on RPI3? (Raspian, Ubuntu...)

 

[luni]

Edit: Saw Kurts answer too late...
----

Serial1.send() seems to split the request into two

Hm, the serial protocol will simply put one byte after the other onto the bus, the time between the bytes is not defined. What exactly do you mean by "splitting"? Do you suspect some additional time after the 8th byte?

On the Raspi-side I get the Buffer.

What do you mean by that? How could the Raspi possibly know when the byte it just read in is the last of the string you sent? I.e. when is the RasPI supposed to stop listening and return the buffer? Usually you need to define some "end of string" character (e.g. char(0) or \n) and you read in bytes from the bus and copy them to some buffer until you get such an end of string character. You then stop reading and convert the buffer to a string. (or do whatever needs to be done with it).

 

[AdmiralCrunch]

Might help to know how Serial1 is connected to your RPI3? Through USART on connector, some USB to serial adapter...

I have connected the RX/TX from the Teensy to the TX/RX-GPIO on the Raspi3

On the Raspi I run Stretch-lite with only xserver, chromium and NodeJS

 

[KurtE]

Then I suspect there is an issue with how NodeJS is working with the data.

If it were me, the first thing I would do is to hook up one of Logic Analyzers up to the RX/TX pins of the Serial1 port and verify everything is correct, which I am sure it is. I would also double check wiring for things like ground... But since you are receiving data I would assume that is correct.

I would then try something else on RPI... Like can I simply receive the data using a simple command line...

Might check first and/or change TTY settings. Something like: stty -F /dev/ttyS0
Sorry I sometimes have to do a few trial and errors, but maybe some command like: cat /dev/ttyS0 | hd

And see what data is returned from your Teensy...

And/Or would hack up a quick and dirty C/C++ program to do it....

But again maybe it is some setting in NodeJS or how you are using it? Maybe there is some form of flow control it is trying to do.

 

[Nominal Animal]

When you read data from an actual serial port, not an USB-serial bridge, the Linux kernel tells an application that is reading data from that serial port that data has arrived at the next earliest opportunity. When that is, depends a bit on the load on the machine, and the kernel tick config, but we're talking about sub-milliseconds to dozens of milliseconds here. During that time, the kernel will buffer any additional received bytes.

In practice, depending on the hardware and serial port settings, each successful read from the hardware serial port yields something like 1 - 15 bytes. (Maybe more on low-power CPUs with very high baud rates.)

There is nothing, absolutely nothing, that says that just because a microcontroller writes N bytes over the serial connection, a single read should return those N bytes, too. All serial port reads may be short, and you must be prepared to use a loop to receive the entire message, if you want to process whole messages.

Also, if the microcontroller sends separate messages in quick succession, it is completely possible for one read to provide the tail end of the former message, as well as the first few bytes of the latter message.

(I'm not sure about Raspberry Pi hardware used here, but if the actual hardware serial port is implemented using an USB bridge, that bridge could be configured to use 8 byte packets, which would explain the 8-byte reads. Essentially, the bridge would cache up to 8 bytes of serial data, and send them to the kernel as a single USB message. Such small message size means the USB bridge chip does not need much internal RAM; it could even be integrated into a helper chip that implements other, much more important functions, like Ethernet for example.)

In C, when I use blocking reads (as opposed to nonblocking I/O with select or poll), I often use a function similar to

size_t  read_bytes(const int descriptor, unsigned char *buffer, const size_t min, const size_t max)
{
    size_t  have = 0;
    ssize_t n;

    if (descriptor == -1 || !buffer || min > max) {
        errno = EINVAL;
        return 0;
    }
    if (max < 1) {
        errno = 0;
        return 0;
    }

    while (have < min) {
        n = read(descriptor, buffer + have, max - have);
        if (n >= 0)
            have += n;
        else
        if (n != -1) {
            errno = EIO;
            return have;
        } else
        if (errno != EINTR)
            return have;
    }

    errno = 0;
    return have;
}

The above function also sets errno unconditionally, so if you expect a message that has a two-byte header, and can be up to 64 bytes long, you can do for example

unsigned char  buf[64];
size_t  len;    

len = read_bytes(serial_port_fd, &buf, 2, sizeof buf);
if (errno) {
    /* Error occurred; abort or handle otherwise */
}

/* You now have len bytes in buf, with len >= 2. */

If you know the next message is exactly 10 bytes long, you can do if (read_bytes(serial_port_fd, &buf, 10, 10) != 10) { /* Error! */ } to read those ten bytes.

This works for serial ports, other character devices, and sockets; they all may return "short" reads or writes.

 

[KurtE]

Yep - on RPI and the like, I wouldtypically open the file, and then use termios to setup the stream. Things like making sure it is N8 and no flow control and no translatings...

 

[AdmiralCrunch]

Hi guys,

first, thank you very much for your answers .. I don't know if I understood it correctly, but I think I found a solution/workaround ..

I now terminate the string beeing send with a ";"
On Raspi-side I check if the ";" is in the sting... if not, the string is beeing concated to a temp-var .. if ";" is recognized in the string, the last part is also beeing concated to the temp-var and then this temp-var is beeing send to my frontend via websockets ..

this seems to work .. I now deal with a other problem but I assume this is a other issue

if I now console.log() the temp-var, I see the full string correctly.. so I send it via websockets to my frontend where the data is beeing appended to HTML.
if I now switch to my frontend, I can see the full data in the HTML.. thats exacly how I want it to be.. now I send some data again from the teensy.. and now again some chars, this time from the beginning from the string, are missing -.- .. wtf .. I switch back to console and see, that also in the console.log()-output those chars are missing ... I send again and look on console... everything ok... switch to frontend.. send again.. chars are missing..

like a monkey I am hanging from one problem to the next one ^^

 

[AdmiralCrunch]

Might check first and/or change TTY settings. Something like: stty -F /dev/ttyS0
Sorry I sometimes have to do a few trial and errors, but maybe some command like: cat /dev/ttyS0 | hd

And see what data is returned from your Teensy...

you are right!!

cat / dev/ttyAMA0 shows the complete strnig coming in ..

what I don't understand is the hd-pipe "| hd" .. i think the fist col is the memory-address, right? .. then there are two cols, this could be the buffer.. but why are there two?

 

[Nominal Animal]

00000000  c6 94 4e 20 f5 8d 3f 01  9a 64 5d 9d e3 91 e3 dc  |..N ..?..d].....|
00000010  bc 39 bd c9 10 d6 55 27  2b 2e 3d 6f 3e f2 5c 35  |.9....U'+.=o>.\5|
00000020  fb df bf e0 b8 63 76 9a                           |.....cv.|

The first column is the offset, in hexadecimal. That is followed by sixteen hexadecimal values, describing the bytes that were received. For ease of reading, there is an extra space between the eight and the ninth byte. At the end of the line, between the pipe characters, is the ASCII representation of the input, but with nonprintable characters (control characters et cetera) replaced with full stops (.).

If you prefer, you can use hexdump -v -e '/1 "%_ad:\t"' -e '/1 "%02X hex"' -e '/1 " %3u dec"' -e '/1 " %03o oct"' -e '/1 " '"'"'%c'"'"'\n"' for one line per input received, showing the byte value in hexadecimal, decimal, octal, and as a printable character. It'll look something like

0:	3C hex    60 dec   074 oct   '<'
1:	41 hex    65 dec   101 oct   'A'
2:	40 hex    64 dec   100 oct   '@'
3:	22 hex    34 dec   042 oct   '"'
4:	67 hex   103 dec   147 oct   'g'
5:	3E hex    62 dec   076 oct   '>'
6:	3A hex    58 dec   072 oct   ':'
7:	3E hex    62 dec   076 oct   '>'
8:	48 hex    72 dec   110 oct   'H'
9:	33 hex    51 dec   063 oct   '3'
10:	5B hex    91 dec   133 oct   '['
11:	6A hex   106 dec   152 oct   'j'
12:	60 hex    96 dec   140 oct   '`'
13:	72 hex   114 dec   162 oct   'r'
14:	3A hex    58 dec   072 oct   ':'
15:	4F hex    79 dec   117 oct   'O'

 

[AdmiralCrunch]

hi,
thank you very much, this is awsome

maybe one last question.. I try to send a variable through serial but at the raspi-side nothing comes in(?)

int c = 0;

void pollShift() {
    if (btnShift.update()) {
      if (btnShift.fallingEdge()) {
          cfg.shiftStatus = true;
          Serial.println("Shift pressed..");          
          Serial1.write("bla:");
          Serial1.write(c);
          Serial1.write(";");#
          c++;
      } else {
        Serial.println("Shift released..");
          cfg.shiftStatus = false;
      }
    }
  }

do I have to take a nap? I don't see whats is wrong here ..oO .. "bla:" and ";" come in, but not the value of c

 

[Nominal Animal]

Serial1.write(c) writes the character corresponding to the value of c, here zero. So, I'd expect you to see

00000000  62 6c 61 3a 00 3b                                 |bla:.;|
00000006

if you look at it at the other end using hd. Note the 00 corresponding to the value of c.

Consider initializing e.g. int c = 'Q'; to see if it makes any difference.

 

[PaulStoffregen]

What makes me curious is, that when I do Serial1.write("blaa:110");, I get the string on Raspi as it it.. in one piece.
But, when I do Serial1.write("blaaa:110");, then I get first request with "blaaa:11" and then comes a second request with "0"

This sort of behavior is a result of the FIFO watermark setting. The UART on Raspberry Pi has a 16 byte FIFO buffer. The Linux kernel driver is almost certainly setting the watermark as 8 bytes. Setting it at half the FIFO size is pretty common, effectively using half the FIFO to reduce interrupt overhead and the other half to allow for interrupt latency without loss of incoming data.

The watermark is the number of bytes the FIFO will receive before it generates an interrupt. UARTs with FIFO always have some sort of timeout too, where an interrupt is generated when any number of bytes less than the watermark setting are in the FIFO.

What's probably happening here is Teensy sends all 9 bytes back-to-back. The Raspberry Pi UART receives them and generates an interrupt after the 8th byte. So the kernel reads the 8 bytes from the UART. Then the 9th byte arrives and sits in the FIFO for a short time, until the UART's timer causes it to generate another interrupt. The kernel then reads the UART and finds 1 byte, and makes it available to user space.

This is how FIFO-based UARTs work. Serial1 and Serial2 on Teensy 3.2, 3.5 & 3.6 work this way too (but the FIFO is 8 bytes, with watermark at 4 bytes). The data appears to arrive in small bursts, which are the FIFO watermark setting. If your system has other stuff hogging interrupts, you'll sometimes see more than the normal burst, because extra bytes arrived between the time the UART generated the interrupt and the kernel (or Serial interrupt on Teensy) is able to actually read the UART and put the data into the memory-based buffer where it'll later be read by the user-level program.

 

[Nominal Animal]

This sort of behavior is a result of the FIFO watermark setting.

I agree completely.

I should have looked at the RasPi UART hardware myself first, and not post vague speculation. The /dev/ttyAMA0 refers to the ARM PL011 PrimeCell UART, whose behaviour is well documented, and the kernel driver sources here. If one ignores my speculation as to why it happens in this particular case, my post should still be useful. Apologies for the noise.

 

[AdmiralCrunch]

I agree completely.

I should have looked at the RasPi UART hardware myself first, and not post vague speculation. The /dev/ttyAMA0 refers to the ARM PL011 PrimeCell UART, whose behaviour is well documented, and the kernel driver sources here. If one ignores my speculation as to why it happens in this particular case, my post should still be useful. Apologies for the noise.

absolutely nothing to apologize for.. in fact, you helped me a lot