Is teensyduino serial monitor somehow special? Problems with other terminal programs.

Status
Not open for further replies.
D

DougMcK

Well-known member
I have a program running fine on a Teensy 3.6. One function is to print a table of values to the USB/serial port for display on the host PC. I cannot ever seem to induce a problem viewing the result with the serial monitor in the Arduino environment (1.8.5 with teensyduino 1.41).
But, if I use another terminal program such as RealTerm, Termite, or Tera Term I quite often get errors in the printed data. These intermittent errors typically show up in the same place, suggesting that it is not entirely random. For example, if I print a 256 row table, each with 15 elements, row 238 will sometimes miss some elements (but never on the Arduino serial monitor).
I have found the problem with different PCs, windows 7 and windows 10, and different cables. I have tried many different baud rates and have not found a way to make the problem better or worse.

The one constant seems to be that the serial monitor never ever has a problem, and three other serial programs all show an intermittent problem at similar place in the data...

I start communications with

Serial.begin(19200);

The code that prints the array is

Code: 
  Serial.println("Data table");
  for (int a=0;a<nRows;a++){
    Serial.print("Row "); Serial.print(a+1); Serial.print(" : ");   
    for (int c=0;c<(nCells);c++){
      Serial.print(Data[arrIdx(a,c)]/scale);Serial.print("  ");
    }
    Serial.println();
  }

the arrIdx function returns an uint32_t and just deals with some housekeeping. In any case this works perfectly with the arduino serial monitor, and mostly ok with other terminal programs.

I feel that something is getting messed up that has nothing to do with this particular fragment of code, but more to do with how serial communications are handled in general.

Any thoughts?
Doug
 
Last edited by a moderator:
Not enough context in that snippet. Is that one time in setup() or in loop()? If in loop() Is there any time constraint on how fast it repeats the print?

If that were in loop() and printing as fast as it can without something limiting the repetition - like delay(20); or some significant amount of processing - then the T_3.6 will keep the output stream to the PC so full MOST terminal programs will be overwhelmed and data display will have dropouts.
 
That snippet is executed once, in response to an external command. So, in testing, at most only once every few seconds.
If I insert a delay(1) after every row of output, it seems to work ok. (Seems a bit clunky though!)
doug
 
Even once with those values for nRows and nCols (not in the code I hadn't looked for them) - 256 x 15 elements with new lines and spacing that will have T_3.6 - even a T_LC - would likely overwhelm the PC with that unceasing batch of output.

It needs to delay or slow down in some fashion except in ideal conditions on the PC side to display that.

Going to TD 1.42 Beta 6 and using the new PJRC TeensySerialMontior should be even faster - but some wait mechanism is needed for reliable transfer.
 
Are you loosing data or does it completely stop at that line?

You want to set fastest priority after serial.begin for testing?
Code: 
  Serial.begin(115200); // usb serial
  [B]NVIC_SET_PRIORITY(IRQ_USBOTG, 0);[/B]
 
Would you like me to investigate?

If so, I need you to give much more specific steps to reproduce the problem. At the very least, I need the complete program to copy into Arduino and upload to a Teensy 3.6. If the program requires other hardware connected to Teensy, I need to know exactly what hardware to buy or build.

If your program is large and you trim it down (which is a very nice thing), please double check that the code you post still does actually exhibit the problem. Obvious as this sounds, we've had plenty of cases where the posted code didn't recreate the issue!

I can test on either Windows 7 or 10. Just say which. Likewise, give me the link to the exact terminal emulator program to install. I have a test machine that I re-image to a clean Windows install for testing, so it will have no software other than Windows when I start the test.

Please also understand I am not a Windows user. I primarily use Linux (and Unix since the late 1980s) and sometimes Macintosh. I've never really used Windows much. I do have a Windows test system here, but I'm not an expert when it comes to Windows. Please be specific about the stuff I should actually do with Windows to recreate the problem.

If I can reproduce the problem, I do have a USB protocol analyzer and other tools to truly investigate exactly what is happening. The hard part is managing to recreate the issue.....
 
When setting the baud rate to 19200 and each character needs 10 bits (including start and stop bits), and thanks to the serial handshaking protocol, you can only transmit around 1760 characters per second. The attempt to send more will likely cause problems, either in the sender (Teensy) or the receiver (Terminal program), depending on how quickly the transmission is acknowledged (which does depend on other factors that the baud rate). So, as long as you don't show screenshots of a serial protocol analyzer, only blind guessing can be done.
 
With USB serial, the baud rate is ignored. The USB protocol implement true end-to-end flow control, so unless something is going very horribly wrong you should never have data loss.

With real hardware serial, as in Serial1, Serial2, etc the baud rate does matter.
 
With USB serial, the PC side controls the baud rate than other traditional serial settings. These aren't used at all by the USB communication.

However, they are communicated by USB to Teensy and can be read using Serial.baud(), Serial.dtr(), etc. You would want to read those if making a USB to serial converter, to configure the actual serial with the settings chosen by the PC. An example showing how this works is in File > Examples > Teensy > USB_Serial > USBtoSerial.

It's theoretically possible, but in reality not ever done, for the PC side software to do something crazy like checking the actual speed of data against the expected speed due to the baud rate setting... and then to something really dumb like discard some of the bytes rather than show them all on the screen.

Other explanations are possible. On non-Windows systems (especially Linux) this sort of problem can happen when 2 programs open the same port. But that is very unlikely on Windows, since Windows doesn't allow more than 1 program to access the device.

Windows 7 is riddled with USB bugs, especially in the serial driver. But usually they result in no data at all, not partial data. This one is the most confusing, and the main reason I recommend using Windows 10 where Microsoft finally fixed this bug.

 
Paul, thanks for your offer to investigate.
This problem is not causing me serious grief right now because it seems the workaround of inserting a delay(1) after each row of output suppresses the problem.
It is, however, disquieting and if you are curious about what's going on I will happily make a minimal version of my program for you to experiment with.
Doug
 
you can also use delayMicroseconds(val) to go faster than waiting for 1ms

example to wait for 50uS
delayMicroseconds(50);

change value higher or lower, to test results
 
Indeed I could. In this particular case there is no problem with the slowdown caused be relatively long delays, and this (ugly) workaround is fine.
I am more interested/concerned about whether there is actually a more pernicious bug somewhere. I think I understood Paul to be saying that this should not happen because of true end-to-end flow control, and I do sometimes have lost/corrupted data.
doug
 
This seems like a Windows App problem getting and parsing and presenting the unending stream from the Teensy.

I've seen this a few times when pushing uncontrolled amounts of data at full speed - over driving output is asking for trouble - depending on the App and computer. Most obvious a year or so back was from a Python sketch that initially was really slow.

IDE Sermon and PJRC's TeensySerMon have improved since a couple years back and my fav TyCommander is a few bytes ahead last I checked (sketch below) - though that didn't show data loss in those three Win Apps - but a reduction in loop()'s per second with sketch below.

Here is a sketch I wrote for another post - and then modified for another - cleaned up a bit for here :
View attachment MaxUsbLC.ino
Click to expand...
>> As posted for 3 seconds it just repeatedly Serial.print()'s and counts how many times it gets to do that in that time.
>> That count is displayed after the 3 seconds for a 'measure of throughput'.

The output when right is a const length line of chars - with the last being counting HEX #'s. It shows no signs of data loss.

And when run on T_3.6 .vs. T_LC the counts change - but not by as much as one would expect - T_LC can keep full USB very well!

More noticeable is moving the Teensy from USB2 HUB to USB3 HUB to USB3 Native port. All show expected output without loss - but the counts change based on USB processing rate on the PC end.

Also visible is a change in the rate of speed when using IDE_SerMon .vs. Teensy_SerMon .vs. TyCommander - subtle but related to the Apps ability to process and present data.

I suspect this sketch will show 'dropped data' run against the indicated: RealTerm, Termite, or Tera Term
 
Well, that's interesting. Running MaxUSB.ino on my surface pro 4 windows 10 PC I record a pretty reliable count of around 83,000 counts if I use the Arduino serial monitor.
The other terminal programs I've tried are less happy. Termite manages 16,000 to 45,000 and Tera Term 16,000 to 98,000 (but mostly at the low end). Real Term seems to hang with a full buffer.

The relative solidity of the Arduino serial monitor program seems to explain why I've not been seeing problems with it, unlike the others. I'm still curious about data loss, though.
 
That isn't an exactly scientifical sketch - it gives a general trend - I've seen anomalous high readings that are probably easily explained / removed - but easier to ignore.

I'm curious what and where you quantifiably see data loss and how to repro and understand it? Teensy working with USB does flow control - reducing the reported loop()'s - but it is up to the PC side app to not get overwhelmed and display what it gets. If a few apps get higher rates and show no data loss - it is within the APP showing lower throughput and loss to work within the system to maintain data integrity.

That sketch shows clear trends in the variability of the PC App to pull, buffer/process, and display data at that rate. And moving that from indicated Port to Hub etc should show a logical trend if the App can keep up.
 
tonton81 said:
Are you loosing data or does it completely stop at that line?

You want to set fastest priority after serial.begin for testing?
Code: 
  Serial.begin(115200); // usb serial
  [B]NVIC_SET_PRIORITY(IRQ_USBOTG, 0);[/B]
Click to expand...

I just added this line and it will perhaps change Teensy behavior - but the problem is on the PC end. When I put it in my sketch it made no difference as nothing else is contending with CPU time and it is already feeding and flushing USB as fast as it can.
 
If I have a failure (not seen on arduino serial monitor, only on other programs) it typically shows up as missing values in the printed table, or missing characters. Sometimes, the transfer just appears to stop, though the Teensy is happy and ready to respond to the next command.
 
DougMcK said:
If I have a failure (not seen on arduino serial monitor, only on other programs) it typically shows up as missing values in the printed table, or missing characters. Sometimes, the transfer just appears to stop, though the Teensy is happy and ready to respond to the next command.
Click to expand...

Those programs are being deficient/defective in their processing on the PC side. The Teensy can push out most all of the ~8 MBytes/ sec the USB can support - that is a lot to process going through an OS to the PC - and it takes efficient and accurate work to keep up.
 
it doesn’t help when the PC slows down to wait for a hard disk to catch up either, go SSD and your OS will be more responsive to other resources
 
If the problem really is only slowness on the PC side (and I'm not 100% convinced that's the only explanation), things will get much, much faster with the next hardware generation where we have 480 Mbit/sec USB rather than 12 Mbit/sec, and 600 MHz Cortex-M7 rather than 100-180 MHz Cortex-M4, and much more powerful & efficient DMA within the USB hardware (entire transfers and arbitrary length data queuing, rather than dealing with single packet per interrupt and only 2-level ping/pong buffering from the hardware).

Today I believe Arduino Due is still the only commonly used board with 480 Mbit/sec USB. But it's software *still* isn't leveraging DMA. For a long time it was slower than Teensy 3.0, despite the 20X faster raw USB speed. It's improved somewhat in recent years, but nothing like the next generation of Teensy boards will do. I'm going to make sure Teensy's next USB stack can easily saturate the entire bandwidth, even if you run stuff like the audio library consuming lots of CPU time... but of course only if the host side keeps up. USB Serial will still be USB bulk protocol, which does end-to-end flow control.

So I'd really like to investigate what's up with these terminal emulators. It's possible they simply suck. Perhaps they're using multithreading (the only way to some things in Windows) and perhaps this data loss is simply a race condition or other thread interaction bug that doesn't get noticed and fixed because so few devices are able to send data at these speeds? Or it could be something more fundamental. Maybe we need Teensy's USB stack to send more zero-length packets?

Again, I'm not a Windows user. To even start digging into this, I need your help with very specific steps on Windows, starting from a perfectly clean no-software-yet Windows system, to reproduce this problem.
 
PaulStoffregen said:
things will get much, much faster with the next hardware generation where we have 480 Mbit/sec USB rather than 12 Mbit/sec, and 600 MHz Cortex-M7 rather than 100-180 MHz Cortex-M4, and much more powerful & efficient DMA within the USB hardware (entire transfers and arbitrary length data queuing, rather than dealing with single packet per interrupt and only 2-level ping/pong buffering from the hardware).
Click to expand...
Paul,
as you brought the theme up, are you already working on it, or still trying to explore the options?
 
Paul, I will see if I can recreate the problem with a minimal version of my program, and then include a link to the problematic terminal emulator(s). It most likely won't be today, though..
doug
 
I got a free license for MegunoLink Pro and it was horribly slow when I tried to use it. Not sure if it has improved since I installed and tried to use some time back. It has a free trial version.

As noted a user had a python interface that was horribly 'lossy' - until it was user optimized based on seeing the data rates Teensy could pump out. So it could be done - just like these other apps though it needed to run against something showing the trouble of too fast - not standard Arduino or limited rate serial terminal devices.

Even TyCommander can be sluggish with the MaxUSB test { seemed to be doing full page draw updates rather than scrolling lines fast } - when it is getting 20,000 line bursts and it was buffering 200,000 lines it was asking Windows for RAM and then discarding it - and also stuffing it to disk. When I changed the setting to buffer only 20,000 lines it stopped being so clunky - which seemed to be then just a bit faster than Teensy Sermon - but the same scrolling display look.
 
Status
Not open for further replies.
Back
Top