Teensy3.0 data loss over Serial/USB problem - solved(?) - likely caused by 96MHz overclock

B

badsector

Well-known member
I did search the forum but didn't find a solution, but some related information about USB.

I have a Teensy3.0 from which I print ASCII data via Serial (which is the USB connection) and capture the data in TeraTerm. I am intermittently losing small chunks of text. I am unsure as to where in the chain the problem lies, because I read that transfer of "serial" data via USB should be 100% reliable. I realise that the teensy OS code creates a "fake" Serial data stream, which is great for ease of use. I use TeraTerm to capture the text, as it is the simplest thing to do.

Of course I have suffered data loss at high baud rates with genuine hardware serial data transfer (via a FTDI PCB to the USB port of a PC) on a different project, but I was able to solve by implementing hardware handshaking (CTS or whichever signal was needed - I forget now). It was for a Mega2560 and I had to hack the OS code to make it work, by using a spare pin to look at CTS.

The project was started to give me a way to exercise Atari ST 80 track 3.5inch floppy disk drives, in order to diagnose faults with the drive(s). I used a Teensy3.0 because it was to hand. Data to/from the floppy interface is buffered with open-collector TTL. After implementing the basic drive control (motor on, head step, etc.) I wanted to see if I could decode the read data stream. I guess it's the same sort of idea as KyroFlux or Greaseweasel, however I am decoding the data stream on the fly, so the Teensy3.0 is operating more like a regular FDC (floppy disk controller) IC. With my latest code additions, I can read a track of 9 or 10 sectors of the standard 512 bytes into a track buffer (byte array).

I implemented a simple user interface to control the Teensy3.0 using single key commands. When reading a sector I print the 512 bytes of the sector in hex and ASCII. This is useful for confirming correct operation and seeing the disk format and contents. One new command reads all 80 tracks (1 or 2 sides) and dumps all sectors. I capture the text in TeraTerm. I have a simple Perl script which extract the ASCII hex bytes and writes a 360k or 720k binary file. This file can be read by Atari ST emulators.

I am now dumping the data from my collection of old Atari ST disks, which I kept from the 80's. I have thankfully recovered much of my old source code for various projects.

The problem is that there are glitches in the captured data log file, where the serial data stream has gone wrong. It happens about once or twice per disk capture. A small section of around 100(?) bytes is missing and it possibly is replaced with some repeated data. I can go re-read any garbled sector data, and then edit the log file manually, but this is tedious and really slows down the task.

The (occasional) glitch occurs when writing all 9 sectors from the sector buffer to Serial (using Serial.print() ), which is done after every track read. I have tried adding "delay(x)" in various places in the print loop. I have tried using Serial.availableForWrite() to see if this gives an indication of buffer overflow, but nothing appears wrong. I'm unsure as to whether this is the right command or perhaps I am using it incorrectly?

One clue was that if during a disk dump, when TeraTerm is displaying scrolling ASCII text, if I select "File->Show Log Dialog" the screen freezes for a moment until the dialog appears. When I review the final log file, there is always a missing chunk of data, larger than the small glitch, but showing that the Teensy3.0 must still be sending data when TeraTerm is unable to receive/buffer.

Are there some other Serial functions which I can use to help me determine whether more text can be printed?
How does the Teensy3.0 "see" that the sink (TeraTerm) is temporarily unable to take any more data?

thank you for any help.

Here is an example of the data loss caused by the "File->Show Log Dialog". Data from sector 7,8,9 and 0 have been lost. If I can understand how to code to prevent this, it will solve the original data loss problem.

*track 5 side 0 sector 7*
61 00 f5 8c 45 f9 00 00 f6 20 61 00 f5 b2 7e 08 61 00 f6 3e b0 44 67 f6 b0 45 67 f2 41 f9 00 00 * a_..E.__. a_..~_a_.>.Dg..Eg.A.__
82 5c 43 f9 00 00 82 68 74 0b 32 19 b0 18 57 ca ff fa 66 da 41 f9 00 00 f6 20 4e fb 10 fe 47 f9 * .\C.__.ht_2_._W...f.A.__. N._.G.
00 00 9c 06 49 f9 00 00 9c 11 22 68 02 c4 22 8b 22 68 02 dc 22 8c b8 7c 00 15 4e 75 61 e0 67 4c * __._I.__._"h_."."h_."..|__Nua.gL
08 a8 00 07 02 a9 67 06 7e 1c 61 00 fa d8 7e 1d 61 00 fa d2 7e 1e 61 00 fa cc 61 56 60 9e 47 f9 * _.___.g_~_a_..~_a_..~_a_..aV`.G.
00 00 9b f0 49 f9 00 00 9b fc 61 be 66 4a 61 00 f0 8e 67 18 08 e8 00 07 02 a9***** STATUS Sector Order = 01:02:03:04:05:06:07:08:09:*****
***** STATUS Header Crc = dac6(dac6):8f95(8f95):bca4(bca4):2533(2533):1602(1602):4351(4351):7060(7060):605e(605e):536f(536f):*****
***** STATUS Sector Crc = 3392(3392):381b(381b):d816(d816):37d8(37d8):053d(053d):91c1(91c1):fd2f(fd2f):4461(4461):5ed5(5ed5):*****
*track 6 side 1 sector 1*
4f 70 3a 20 50 72 69 6e 74 20 74 65 78 74 00 49 4f 70 3a 20 4d 6f 76 65 20 62 6c 6f 63 6b 00 49 * Op: Print text_IOp: Move block_I
4f 70 3a 20 43 6c 65 61 72 20 62 6c 6f 63 6b 00 49 4f 70 3a 20 53 61 76 65 20 62 6c 6f 63 6b 20 * Op: Clear block_IOp: Save block
61 73 2e 2e 2e 00 49 4f 70 3a 20 50 72 69 6e 74 20 62 6c 6f 63 6b 00 49 4f 70 3a 20 4f 70 65 6e * as..._IOp: Print block_IOp: Open
20 66 69 6c 65 2e 2e 2e 00 49 4f 70 3a 20 4d 65 72 67 65 20 66 69 6c 65 2e 2e 2e 00 49 4f 70 3a * file..._IOp: Merge file..._IOp:
20 52 65 65 6c 69 6d 69 6e 61 74 65 20 74 65 78 74 00 6e 6f 20 69 63 6f 6e 20 6f 70 65 72 61 74 * Reeliminate text_no icon operat
69 6f 6e 73 00 49 4f 70 3a 20 4d 6f 76 65 20 69 63 6f 6e 00 84 8e 94 99 81 9a 4c 6f 61 64 00 53 * ions_IOp: Move icon_......Load_S
61 76 65 00 4e 65 77 20 70 61 67 65 20 6c 65 6e 67 74 68 3a 00 4e 65 77 20 6c 69 6e 65 20 6c 65 * ave_New page length:_New line le
6e 67 74 68 3a 00 54 6f 20 70 61 67 65 3a 00 54 6f 20 6c 69 6e 65 3a 00 5b 33 5d 5b 53 79 6e 74 * ngth:_To page:_To line:_[3][Synt
61 78 20 65 72 72 6f 72 21 5d 5b 43 61 6e 63 65 6c 5d 5b 33 5d 5b 4e 75 6d 62 65 72 20 3e 20 32 * ax error!][Cancel][3][Number > 2
35 35 20 28 24 31 30 30 29 20 21 5d 5b 43 61 6e 63 65 6c 5d 5b 33 5d 5b 4d 75 6c 74 69 70 6c 65 * 55 ($100) !][Cancel][3][Multiple
20 64 65 66 69 6e 69 74 69 6f 6e 21 5d 5b 43 61 6e 63 65 6c 5d 5b 33 5d 5b 49 6e 73 74 61 6c 6c * definition!][Cancel][3][Install
61 74 69 6f 6e 20 74 6f 6f 20 62 69 67 21 5d 5b 43 61 6e 63 65 6c 5d 50 52 45 41 4d 42 4c 45 00 * ation too big!][Cancel]PREAMBLE_
50 4f 53 54 41 4d 42 4c 45 00 45 4e 44 4c 49 4e 45 00 42 45 46 4f 52 45 4c 49 4e 45 00 42 45 46 * POSTAMBLE_ENDLINE_BEFORELINE_BEF
4f 52 45 50 41 47 45 00 41 46 54 45 52 50 41 47 45 00 3a 3d 00 43 72 6f 73 73 20 72 65 66 65 72 * OREPAGE_AFTERPAGE_:=_Cross refer
65 6e 63 65 20 6c 69 73 74 20 66 6f 72 20 03 00 04 20 54 6f 74 61 6c 3a 20 31 32 33 34 35 36 37 * ence list for ___ Total: 1234567
38 00 00 44 49 53 4b 20 44 52 49 56 45 00 50 52 49 4e 54 45 52 00 42 4c 4f 43 4b 20 52 45 47 49 * 8__DISK DRIVE_PRINTER_BLOCK REGI
*end*
*loop*
Click to expand...
 
Teensy 3 is slow compared to a T_4.x - but for that PJRC refined the IDE SerMon using Teensy Ports to be more faithful.

Give that a try instead of TerraTerm and see if the same loss occurs. The problem is doubtfully the Teensy sending, but rather on the receiving end and faithfully doing a GUI display of the data in a timely fashion. Other than TyCommander (forum contribution by @kromix) a few other terminal programs used failed to keep up even with older T_3.x's rate of output.

One other option for testing might be to add delayMicroseconds( 1 ) every 'so many' characters perhaps 64, or at some natural breaking point in the code 'between lines or groups'.
 
Thank you for the reply.

I have been trying various things and I have got very confused. Sorry of the text below is unclear. I am updating it as I try more code changes.

I had a moment when I thought that I had figured out what was wrong. I hadn't selected "hardware" flow control in TeraTerm. Unfortunately enabling that made no difference.

NOTE: When TeraTerm hardware flow control is used with data sent from genuine Arduino hardware serial port via a FTDI USB-to-serial board, this does work in capturing data with no loss even at very high baud rates (as long as you control CTS/DTR at the Arduino end of course!).

I took at look at the USB code in ".../Arduino/hardware/teensy/avr/cores/teensy3/..." but it went completely over my head.

If I understand correctly the Teeny3.0 USB hardware is used to implement a FTDI "like" device. Again I assume it is not at all connected with any of the genuine hardware serial ports? I assume that it has a VIP and PID and must have a driver installed on the PC to support it? Is this how Teensy3 "Serial()" works?

In trying to diagnose the data loss problem I have made things worse. I did have various delay()'s scattered around the Serial.print() code, but that was slowing down the time to read the disk. I was still thinking like as if I was using the serial hardware port and allowing time for the TX buffer to flush. So instead I used "Serial.availableForWrite()" at various places in the "dumpSector()" routine which does the hex and ASCII 512 byte dump. Now whether I use TeraTerm or the built-in serial monitor, there is a lockup of the serial stream during the sector dump code (quite a way into the disk dump). I am now stumped, because I have no idea how to diagnose the lock-up. Sometimes during a disk capture it locks up, sometimes it is OK.

I have now tried the built in serial monitor and seem the exact same data loss/corruption as seen in TeraTerm. It's not obvious but possibly about 58 data bytes are corrupted (i.e. replaced with incorrect data) shown in bold below. Does this number mean anything, e.g. USB buffer size? Some of the corrupted data is a repeat from the previous line. One part of the corrupted data came from the start of the serial log (13 sectors earlier).

2f 2e 00 0a 4e b9 00 00 04 f6 df fc 00 00 00 0a 3d 40 ff f8 66 00 00 20 4e b9 00 00 9c fa 30 2e * /.__N.___...____=@..f__ N.__..0.
ff fa 48 c0 d1 ae 00 0a 30 2e ff fa 91 6e 00 0e 4a 6e 00 0e 66 a4 4e b9 00 00 9c fa 2e b9 00 02 * ..H...__0....n__Jn__f.N.__....__
0b 66 3f 3c 00 20 4e b9 00 00 01 aa 54 8f 2e ae ff fc 4e b9 00 00 bc ce 4a 6e ff f8 67 00 00 0a * _f?<_ N.___.T.....N.__..Jn..g___
*end*
*loop*
* _f?<_ N.___.T.....N.__..Jn..g___
oot entry count 4 ]] 3c ff ff 60 00 00 08 42 40 60 00 00 02 4e 5e 4e 75 42 a7 3f 3c 00 20 4e 41 5c 4f 2f 00 3f 3c * 0<..`___B@`___N^NuB.?<_ NA\O/_?<
00 20 61 00 00 4a 4e 41 5c 4f 42 67 4e 41 4a fc 4e 56 00 00 48 e7 1f 1c 42 a7 3f 3c 00 20 4e 41 * _ a__JNA\OBgNAJ.NV__H.__B.?<_ NA
5c 4f 2f 00 3f 3c 00 20 72 28 d2 b8 04 ba b2 b8 04 ba 6c f6 3e 2e 00 08 eb 4f 61 00 00 2c 56 c0 * \O/_?<_ r(.._..._.l.>.__.Oa__,V.
48 80 4e 41 5c 4f 4c df 38 f8 4e 5e 4e 75 7e 00 61 00 00 16 72 28 d2 b8 04 ba b2 b8 04 ba 6c f6 * H.NA\OL.8.N^Nu~_a___r(.._..._.l.
Click to expand...


Another experiment. I have enabled hardware flow control in TeraTerm and added the code "while(!Serial.dtr()) { delayMicroseconds(1000); }" in the dumpSector() routine, but DTR is never seen activated even if I pause the TeraTerm scrolling by using it's menu. Just now looking in "usb_seremu.h" the dtr() function appears to be hardcoded to always return '1'.

Summary.

When there is no lockup, I see the same type of data corruption in the built-in serial monitor as was seen originally in TeraTerm.
Sometimes the serial stream will lock-up (does not depend on which serial monitor is used).
Use of SerialavailableForWrite() makes no difference.
Use of Serial.dtr() makes no difference.
 
Odd, this is not a common problem.

Can a simple sketch be posted that demonstrates the issue?

With USB Serial, PJRC has implemented a true USB device using true USB hardware on the Teensy MCU that conforms to the Windows 10&11 built in driver.

<edit> this is Teensy 3? so, USB connect to Host is 12 Mbps - it should not have the ability to saturate a PC with efficient GUI handling - like the PJRC SerMon teensy_ports.

Not sure what the sketch looks like and how fast and continuous the data from Teensy to PC is - and if it is approaching this saturation level?

If not saturating with that much data then the PC or Cable seems to be an issue - though the USB data is error checked to some degree so an iffy cable would just fail to get a usable connection or data?
 
Thank you once again for your reply.

Yes I can make a test sketch by removing all the code to do with floppy hardware control and read stream decode. I already fill the track buffer with default values before doing the read, so that any missing data can be easily spotted. I can leave in the track buffer transfer.

I have quickly hacked out most of the hardware and unrelated functions, but I would like to tidy up a little more. On the first try, because there is no delay due to the track read, the data "screams" up the TeraTerm window. I captured the ASCII data in a log file and ran my Perl script. TBH to my surprise there was the typical small data loss which I commonly see. Only there is only once instance of "corruption" which again is what I regularly see.

99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 * ................................
99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 * ................................
99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 * ................................
*end*
*loop*
* ................................
tes Per Sector 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 * ................................
99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 * ................................
99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 * ................................
99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 99 * ................................
Click to expand...

The
* ................................
Click to expand...
part of the corrupted data (after the word "loop") appears to come from the previous line. Again this is what I have seen previously.

The
tes Per Sector
Click to expand...
part of the corrupted data comes from the header at the start of the log (256k characters earlier - calculated using wc -c). Hmmm that number seems a bit suspicious. It is part of a text string stored in Flash ROM (by default on the Teensy3, i.e. I haven't used F() )..

I will tidy up the debug sketch a little more, do some more testing, then post it with results.

Knwing my luck there will be a simple bug in my code and I will look an idiot! Nevertheless I will be happy to look like an idiot and have working code :)
 
Here is the stripped down code. All (floppy) hardware references have disappeared. What is left is the transfer of 720k of data in hex/ASCII format.

The function that generates most of the text is "dumpSector()" which I have tried to leave as much as possible.

There is a single #define to change the code. TRY1 attempted to use Serial.dtr() which (as mentioned in my previous post) appears to always return '1' for USB serial (AFAIK). TRY2 attempts to monitor the TX buffer using Serial.availableForWrite().

I see corrupted captured data both in TeraTerm and also the built in Serial Monitor.

Finding where the data is corrupted is not easy, so I will need to post my Perl code which converts from ASCII hex to a 720k binary floppy image file, but does various checks as it processed the data. Consider all code to be "quick and dirty", in other words it isn't polished for release (I am the only user).

C: 
//
// Badsedctor example sketch to demonstrate serial data (over USB) corruption.
// Likely caused by inadequate flow control coding?
// Stripped down original code, removed all hardware related code.
//
// Teensy 3.0
//
// I/O = 3.3V  (needs open collector interface to/from floppy)
// CPU = Cortex-M4 @ 48MHz (need to select 96MHz for the CPU clock)
// RAM =  16k bytes
// ROM = 128k bytes
// FTM timer = 2
// PIT timer = 4
//


#include <stdio.h> // for function sprintf
#include "Arduino.h"


//============================CONSTANTS================================


#define ESC ((char)(27))


#define MAX_BPS           512
#define MAX_SPT            10
#define MAX_TRACKS         80  // for Atari DD


//==========================GLOBAL VARIABLES=======================


char sbuf[99];       // used for sprintf


uint8_t track_buffer[MAX_SPT+1][MAX_BPS];
uint8_t sector_buffer[MAX_BPS];


uint16_t bps=MAX_BPS; // HARD CODED FOR DEBUG
uint16_t spt=9;       // HARD CODED FOR DEBUG
uint16_t nheads=2;    // HARD CODED FOR DEBUG


// Sectors are numbered 1 to MAX_SPT
uint8_t  listSector[MAX_SPT+1];
uint16_t listHdrCRCexp[MAX_SPT+1];
uint16_t listHdrCRCact[MAX_SPT+1];
uint16_t listDatCRCexp[MAX_SPT+1];
uint16_t listDatCRCact[MAX_SPT+1];
uint8_t  listTrackIdx;
uint16_t hdr_crc_errors=false; // DEBUG
uint16_t dat_crc_errors=false; // DEBUG
bool     disk_status=false; // DEBUG


//==================================================================


void setup() {
  while (!Serial);
  Serial.begin(500000);
  Serial.println("Atari ST Floppy Disk Reader Program");
  Serial.println("Ready...");
}//end setup


//--------------------------------------------------------------------------------------------




//--------------------------------------------------------------------------------------------


// TRY1 -> I suspect that Serial.dtr() always returns 1, so essentially no flow control or buffer checking at all
// TRY2 -> tries to use Serial.availableForWrite() to prevent transmit buffer overflow


#define TRY2


////////////////////////////////////
// CRITICAL - CAUSES TRANSFER BUG //
////////////////////////////////////


// In an attempt to debug I tried using Serial.send_now() to regularly flush the USB/serial buffer


void dumpSector(void) {
  //char s[33];
  char s[256];
  int sp=0;
  bool dtr=false;
  int av = Serial.availableForWrite(); sprintf(sbuf, "*serial available 1 = %d ",av); Serial.println(sbuf);


  // the hex dump will be 16 lines of 32 bytes, printed as 32 hex characters followed by the equivalent 32 ASCII characters
  for (uint16_t i=0;i<MAX_BPS;i++) {
#ifdef TRY1
    while(!Serial.dtr()) { dtr=true; delayMicroseconds(1000); }
#endif
#ifdef TRY2
while (Serial.availableForWrite() < 32) { delayMicroseconds(1000); }
#endif
    uint8_t b=sector_buffer[i]; sprintf(sbuf, "%02x ",b); Serial.print(sbuf); Serial.send_now();
    if (b>=32) { if (b<0x7f) { s[sp++]=b; } else { s[sp++]='.'; } } else { s[sp++]='_'; } // form ASCII data in s[] array
#ifdef TRY2
while (Serial.availableForWrite() < 32) { delayMicroseconds(1000); }
#endif
    if (((i+1)%32)==0) {
      // print the ASCII data, and reset the index into the buffer for the next line
      s[sp]=0; sp=0; sprintf(sbuf,"* %s %s",s,dtr?"DTR":""); Serial.println(sbuf); Serial.send_now();
      dtr=false;
    }
  }
  Serial.send_now();
}


//+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


bool active = true;
byte k = 25;
unsigned char cmd;


void loop() {
 
  if (Serial.available()) {
    cmd = Serial.read();
    if (cmd=='h') {
      Serial.print(ESC); Serial.print("[2J");
      Serial.print(ESC); Serial.print("[H");
      Serial.println(F(" Floppy_Control implements single character commands to operate the floppy drive "));
      Serial.println(F(" hit the 'g' key to start the serial data debug serial transfer "));
    }


    ////////////////////
    // transfer debug //
    ////////////////////
    else if (cmd=='g') {


      //
      for (uint8_t track=0; track<MAX_TRACKS; track++) {
        // move heads to track
        sprintf(sbuf, "***** TRACK  = %d *****", track); Serial.println(sbuf);
        // clear tracker buffer
        for (uint8_t s=0;s<(MAX_SPT+1);s++) for (uint16_t i=0;i<MAX_BPS;i++) track_buffer[s][i]=0x99;
        //
        for (uint8_t side=0; side<nheads; side++) {
          sprintf(sbuf, "***** SIDE   = %d *****", side);  Serial.println(sbuf);
          while (Serial.availableForWrite() < 63) { delayMicroseconds(1000); }
          //
          // do the track read (REMOVED FOR DEBUG) - replace with delay
          delay(50);
          //
          for (uint8_t i=0;i<(MAX_SPT+1);i++) {
            listSector[i]=99;
            listHdrCRCact[i]=0x9999;
            listHdrCRCexp[i]=0x9999;
            listDatCRCact[i]=0x9999;
            listDatCRCexp[i]=0x9999;
          }
          listTrackIdx=0;
          //
          //
          sprintf(sbuf, "***** STATUS Sector Order = "); Serial.print(sbuf);
          for (uint8_t i=0;i<spt;i++) {
            if ((listDatCRCact[i] != listDatCRCexp[i])||(listDatCRCact[i] != listDatCRCexp[i])) {
              sprintf(sbuf, "!%02d!",listSector[i]); Serial.print(sbuf);
            } else {
              sprintf(sbuf, "%02d:",listSector[i]); Serial.print(sbuf);
            }
          }
          sprintf(sbuf, "*****"); Serial.println(sbuf);
          //
          sprintf(sbuf, "***** STATUS Header Crc = "); Serial.print(sbuf);
          for (uint8_t i=0;i<spt;i++) {
            if (listHdrCRCact[i] != listHdrCRCexp[i]) {
              sprintf(sbuf, "!%04x/%04x/!",listHdrCRCact[i],listHdrCRCexp[i]); Serial.print(sbuf);
              hdr_crc_errors++;
            } else {
              sprintf(sbuf, "%04x(%04x):",listHdrCRCact[i],listHdrCRCexp[i]); Serial.print(sbuf);
            }
          }
          sprintf(sbuf, "*****"); Serial.println(sbuf);
          //
          sprintf(sbuf, "***** STATUS Sector Crc = "); Serial.print(sbuf);
          for (uint8_t i=0;i<spt;i++) {
            if (listDatCRCact[i] != listDatCRCexp[i]) {
              sprintf(sbuf, "!%04x/%04x/!",listDatCRCact[i],listDatCRCexp[i]); Serial.print(sbuf);
              dat_crc_errors++;
            } else {
              sprintf(sbuf, "%04x(%04x):",listDatCRCact[i],listDatCRCexp[i]); Serial.print(sbuf);
            }
          }
          sprintf(sbuf, "*****"); Serial.println(sbuf);
          //


          ////////////////
          // DUMP TRACK //
          ////////////////
          for (uint8_t sector=1; sector<=spt; sector++) {
            while (Serial.availableForWrite() < 64) { delayMicroseconds(1000); } // ATTEMPT AT FLOW CONTROL
            for (uint16_t b=0;b<MAX_BPS;b++) { sector_buffer[b] = track_buffer[sector][b]; }
            sprintf(sbuf, "*track %d side %d sector %d*",track,side,sector); Serial.println(sbuf);
            dumpSector();
            sprintf(sbuf, "*end*"); Serial.println(sbuf);
            sprintf(sbuf, "*loop*"); Serial.println(sbuf);
          }


        } // side
        if (hdr_crc_errors > 0) { sprintf(sbuf, "***** track %d header %d CRC errors *****",track,hdr_crc_errors); Serial.println(sbuf); disk_status=false; }
        else { sprintf(sbuf, "*"); Serial.println(sbuf); }
        if (dat_crc_errors > 0) { sprintf(sbuf, "***** track %d sector %d CRC errors *****",track,dat_crc_errors); Serial.println(sbuf); disk_status=false; }
        else { sprintf(sbuf, "*"); Serial.println(sbuf); }
      }   // track
      // final status
      if (disk_status) {
        sprintf(sbuf, "* disk read with NO errors "); Serial.println(sbuf);
      } else {
        sprintf(sbuf, "* CRC errors found "); Serial.println(sbuf);
      }
    }


    else {
        sprintf(sbuf, "### FLOW CONTROL %d",cmd); Serial.println(sbuf);
    }
    //==========================================================================//


  }


}
 
Perl code to create the binary from the hex ASCII log capture.

Execute with something like: /path/perl hex_2_bin.pl -i CAPTURE.log

Perl: 
#!/usr/bin/perl


#use strict;


print "#########################\n";
print " Simple hex to bin v1.01\n";
print "#########################\n";
if ($ARGV[0] =~ /-version/) {
  return 0;
}


#if(!@ARGV) {
#  print "No Arguments supplied\n";
#  print "-i <filename> \n";
#  print "-o <filename> \n";
#  exit;
#}




while($Arg = shift(@ARGV)) {
  if( $Arg !~ /^-/) {
   # We expected a switch
   print "Switch expected instead of $Arg\n";
   exit;
  }


  # Remove switch
  $Arg =~ s/^-//g;


  if($Arg eq "") {
   print "'-' cannot be used on its own\n";
   exit;
  }


  # examine switches
  if("help" =~ /^$Arg/i || $Arg =~ /\?/)
  {
   print "Help Requested\n";
   exit;
  }


  if($Arg =~ /^i$/i) {
    $i = shift(@ARGV);


    if(!defined($i) || $i =~ /^-/) {
      print "-i switch requires 1 argument\n";
      exit;
    }
    print "input file = ${i}\n";
  }


}


if (!defined($i)) {
  print "input file must be specified using -i switch\n";
  exit;
}


unless(open(IN,"<".$i)) { die("Could not open input file $i\n"); }


$h = '[0-9a-f]';


print "STARTED\n";
my @bin_data;
my $idx=0;
my $lineno=0;
my $lines=0;
my $track;
my $side;
my $sector;
my $waiting=1;


while (<IN>) {
  chomp($txt=$_);
  #print "TXT:$txt\n";
  $data_text = $txt;
  $lineno++;
  #
  if ($data_text =~ /^\*\*\*\*\* TRACK  = 0 \*\*\*\*\*/) { $waiting=0; }
  if ($data_text =~ /^\* disk read with NO errors/) { break; }
  if ($waiting == 1) { next; }
  #
  if ($data_text =~ /^\*track (\d+) side (\d) sector (\d+)/) { $track=$1; $side=$2; $sector=$3; $lines=0; }
  if ($data_text =~ /^\*end\*/) { if ($lines != 16) { print "Block wrong count=$lines track=$track side=$side sector=$sector\n\n"; exit(); } }
  #
  if ($data_text !~ /^\*/) {
    $lines++;
    for ($i=0; $i<32; $i++) {
      #
      if ($data_text =~ /^($h{2})/) {
        $data_text =~ /^($h{2})\s*/;
        $data_text = $';
        $value = hex($1);
        #printf("%02x",$value);
        $bin_data[$idx++] = $value;
      } else  {
        print "ERROR $i at line $lineno ($txt/$data_text)\n";
        #exit();
        $i=32;
      }
    }
    #print "\n";
  } else {
    #print "COMMENT($txt)\n";
  }
}


printf("Total bytes read = %d\n",$idx);


if ($idx == 737280) {
  print "Found Atari 80 track, 9 sectors per track, double sided disk\n\n";
} elsif ($idx == 368640) {
  print "Found Atari 80 track, 9 sectors per track, single sided disk\n\n";
} elsif ($idx == 409600) {
  print "Found Atari 80 track, 10 sectors per track, single sided disk\n\n";
} elsif ($idx == 819200) {
  print "Found Atari 80 track, 10 sectors per track, double sided disk\n\n";
} else {
  print "WARNING: wrong number of bytes for Atari 80 btrack single/double sided disk\n\n";
}


close(IN);


$filename = "out.st";
open DATAOUT,">$filename" or die "Error opening output file $filename: $!\n";
# set the stream to binary mode
binmode DATAOUT;
print DATAOUT pack('C*',@bin_data);
close DATAOUT or die "Error closing file $fielname: $!\n";
 
After some more experiments, I might have found a reason for the problem.

Due to needing to decode the floppy data stream "on the fly" I was over-clocking the Teeny3 to 96MHz.

I experimented with compiling for a CPU speed of 48MHz. I captured data from the debug program three times, with no corruption, although on one run there was a lock-up and the data stream just stopped.

I re-compiled for the CPU overclock speed of 96MHz and made a new capture log. It was corrupted.

So, I'm going to suggest that the data loss problem over USB is likely caused by running the device "out of spec". I am aware that if you over-clock any chip, then some parts will work happily at the overclock speed, but some parts, might not. Perhaps the USB part of my Teensy3 just can't work at 96MHz under some critical conditions of it's state machine.

The danger with over-clocking is that everything might appear to be working well, but you can never be sure that one of the millions of internal signals isn't reaching it's destination in time (at the overlock speed) and sometimes this results in a critical and obvious fail, sometimes it doesn't matter. and sometimes the problem manifests in very subtle difficult to detect ways.

First I'm going to see if my code will run at 48MHz in the full app, but it could be too slow.

I'm also going to find a different Teensy, one which can run at 96MHz (or higher), and try again.
 
Good find on 48MHz testing. The USB Hardware on Teensy is the processor itself.

Perhaps there is a place where a small delayMicroseconds() slowing the output stream would prevent the corruption? Is this printing just for debug testing? I ran into using such a delay writing T_4.x code that wrote code outputting it to SerMon and was seeing corruption when that collected 'data/Code' was then put back into the sketch to build.

If the Teensy is indeed a 3.0 versus a newer larger memory T_3.1 or 3.2 pushing to 96 MHz might indeed have found a way to overwhelm it with the current process.
 
The ASCII printing of the data from the floppy in 512 byte chunks as hex and ASCII data is how I collect the floppy data, so is critical to the application. The same Perl script as posted above processes the log, extracts the hex numbers and converts then into a 720k byte binary image of the floppy which can be loaded directly into various emulators.

I was pleased to find lots of old code (on unmarked floppies) which I had written in the 80's and had considered lost.

The project was not initially designed to read the floppy, just exercise it to allow diagnosis of faults with a couple of Atari ST floppies, but progressed, out of the challenge of "can it be done" to use a micro to mimic a floppy disk controller IC. Using serial was the quick and dirty way of displaying the data read from the floppy. There was the bonus that it was easy to review the ASCII log for the various disks where CRC errors were reported, and by editing the ASCII, generate an image.

I did consider wiring up a SD card module, but that was more effort and coding.

It might seem a strange way to "image" Atari ST floppies, but I did consider other work flows, and even tried some, but I didn't find any way as simple as the Teensy based approach.
 
Very cool recovering and reading that old data! Especially being able to use it again.

Teensy 3 USB uses 64B blocks. If you can tolerate/arrange read delays as needed put a delayMicroseconds(100) between 64Byte writes. That is only 12 blocks for those blocks, so it won't slow things down much. It may need to be 1,000 us given the lower transfer rate of the 12 Mbps USB on the Teensy 3?

Given the limited RAM on the T_3.0 the 96 MHz may be trying to move more data faster than the USB can safely manage.
 
Some additional information regarding USB/Teensy3.X/overclock.

I replaced the Teensy3.0 board with a Teensy3.1. I re-ran the "serial USB transfer debug" program (all hardware code removed).

At 96MHz overclock I got random lock-ups. Serial stopped working completely. Serial worked but froze when transferring track 12/track 17. So serial over USB worked for a while then randomly locked up.

At 72MHz stock speed, serial over USB ran flawlessly for the three time I ran a test. I also opened the "show log dialog" in TeraTerm which causes the TeraTerm scrolling data to freeze. This had ZERO impact on the captured data.

I briefly tried 120MHz overclock and serial froze almost immediately.

Just FYI, YMMV, it could of course be my specific setup. Teensy connects to a small 4 port hub, which connects to the monitor, which in turn connects to the PC. This arrangement is down to cable lengths and location. I have used FTDI USB devices and USB memory sticks in the same port with no issues. Also Teensy3 without overclock works happily in this USB arrangement.

Since other uses might use Teensy3.X serial over USB, I though the possible USB overclock issue worth reporting.

¯\_(ツ)_/¯​

 
OK, so I now have a working system (work flow) using the Teensy3.0 (with the needed 96MHz overclock).

I searched around and found a FDTI board, the common red type, and likely a Chinese clone/fake.

I played around with TeraTerm and flow control, but it was another rabbit hole. The FDTI board has CTS and DTR, but I think TeraTerm only supports CTS/RTS. I did sort of get flow control from TeraTerm to the Teensy working with a different terminal program. I eventually upgraded TeraTerm to the latest version. Hardware flow control now supports CTS/RTS and DTR/DSR. I tried to send one million characters to the TeraTerm at 2M baud, but I never saw DTR change.

I connected the FDTI board to the hardware serial port RX1/TX1 on digital pins 0/1. I settled on 2M baud. It was a quick change to modify dumpSector() to write to Serial1, and associated header println code.

Transfer with no flow control now appears to work 100%, after imaging a number of disks.

I modified the sketch to output the expected sector CRC and modified the Perl script to re-calculate. Other sanity checks were added. This now gives me super high confidence that if there is any serial data corruption, one of the many checks will catch it.

Rather bizarrely, and unreproducible, during one early capture, a string containing a stream of "e5 e5 e5 e5 e5 e5" chars (from an empty sector), had one of the 'e's change into a space. Serial transfer, don't you just love it :) 😩 !!!
 
You must log in or register to reply here.
Back
Top