flush incoming midi buffer at end of setup

snowsh · Nov 3, 2023

i am using usbMidi on a t4.1.

at the end of my setup function i call:
usbMIDI.read(); // flush out any midi that may be in the buffer

This had been working fine. I just built a new PC and since, this does not do what I expect.

the unwanted behavior is:
At the end of startup, it calls:
usbMIDI.read(); // flush out any midi that may be in the buffer

when we reach loop, there is a rush of midi clock data triggering my project. I'm mainly reading the midi clock.

Has there been a change to usbMidi that may be the cause?

I want to clear the incoming midi buffer at the end of setup so any incoming midi data received during the setup phase is ignored.

I initiate USB midi early on in setup in order to make connection to my DAW and send an all notes off command to settle things down.

Any ideas?

joepasquariello · Nov 3, 2023

I can't help on the MIDI stuff, but you probably should tell us what is your platform, IDE, and TeensyDuino, and show your code if you can.

PaulStoffregen · Nov 4, 2023

Maybe more than 1 message is already in the buffer? You probably should try something like this:

Code:

  while (usbMIDI.read()) {
    // ignore incoming messages
  }

snowsh · Nov 4, 2023

PaulStoffregen said:
Maybe more than 1 message is already in the buffer? You probably should try something like this:

Code:

while (usbMIDI.read()) { // ignore incoming messages }

Oh Paul when I saw this reply I thought how stupid of me, so obvious and I was so hoping this was the fix...

Code:

  while (usbMIDI.read())        // flush out any midi that may be in the buffer
  {
    // ignore incoming messages
  }

Unfortunately its still doing this rush. My guess is it has something to do with the interrupt that is called on incoming midi clock. I'm using usbMIDI.setHandleRealTimeSystem(RealTimeSystem). This is my midi setup function called from setup():

Code:

void setupUsbClientMidi()
{
  Serial.print(F("setup midi stuff"));


  usbMIDI.setHandleRealTimeSystem(RealTimeSystem);


  usbMIDI.setHandleSongPosition(mySongPosition);                          // used to set a start point based on where the daw is set at start

  usbMIDI.setHandleNoteOn(myUsbClientNoteOn);    
  usbMIDI.setHandleNoteOff(myUsbClientNoteOff);


  usbMIDI.setHandleSystemExclusive(mySystemExclusiveChunk);

  
  Serial.println(F(" - setup midi stuff done"));
}

Its strange because I have been working on this now for over 3 years, and its only in the last few weeks this rush of clock triggers has started happening. I had the same problem when i started, and usbMIDI.read() fixed the issue. One thing i have added since is usbMIDI.setHandleSystemExclusive(mySystemExclusiveChunk) but i have already disabled it to check and the issue persists. The rush of clock triggers is annoying, even though its over in a split second. Interestingly it reveals the impressive speed of the t4.1 and the processing overhead i still have available i could probably run this project at insane BPM speeds, but then the DAW will likely overload. My original thinking of the being a change to a library is still high in my thinking... Is there a reason why the use of the interrupt handler for real-time system may be linked to this rush even when calling usbMIDI.read() in a while loop? One other thing, I no longer need to go into my DAW (reaper) to reconnect the teensy midi device after a restart - I had put this down to a newer version of reaper on my new PC actually working with active sensing maybe, but then this is not called during the setup at present - I will address that....

Code:

void usbMidiActiveSensingHandler()                                // might prove useful
{
  if (enableActiveSensing)
  {
    if ((currentMillis - lastUsbMidiActiveSensingRefreshTime) > usbMidiActiveSensingDelay)
    {
      lastUsbMidiActiveSensingRefreshTime = currentMillis;

      usbMidiActiveSensing();
    }
  }
}

void usbMidiActiveSensing()
{
  usbMIDI.sendRealTime(usbMIDI.ActiveSensing);
  // TODO: add USBMidiHost, MIDI and MIDI2

}

I appreciate this is going to be tricky to help me with without posting more code, but the project is enormous...

Memory Usage on Teensy 4.1:
FLASH: code:403152, data:168936, headers:8516 free for files:7545860
RAM1: variables:129440, code:392968, padding:248 free for local variables:1632
RAM2: variables:22784 free for malloc/new:501504
EXTRAM: variables:817312

Yes I have some work to do to reorganise memory to free up more space in the stack

but other than this rush issue when loop starts, my project is rock solid and happily runs without any memory crash

joepasquariello said:
I can't help on the MIDI stuff, but you probably should tell us what is your platform, IDE, and TeensyDuino, and show your code if you can.

in answer, win10, IDE arduino 1.8.16, teesyduino 1.58

joepasquariello · Nov 4, 2023

snowsh said:
in answer, win10, IDE arduino 1.8.16, teesyduino 1.58

The gcc toolchain was updated to 11.3.1 in TD 1.58, and the 1.59 beta versions fix some bugs related to class initialization due to the newer C++ standard of the tools. It might be worth updating to Arduino 1.8.19 (latest) and TeensyDuino 1.59b3, just to be sure that's not an issue.

snowsh · Nov 4, 2023

did a quick experiment to see if flushing on the first loop rather than in setup makes any difference

Code:

bool firstLoop = true;        // to track the first time we run loop after setup (global)

void loop()
{
  if (firstLoop)
  {
    while (usbMIDI.read())        // flush out any midi that may be in the buffer
    {
      // ignore incoming messages
    }
    firstLoop = false;
  }

// rest of loop code

}

no good. Still responding to the buffered midi clocks received from the DAW during setup(). My clock counter gets to around 4 measures, 4 beats after the "rush" which lasts maybe 1/10 of a second. So that's allot of clock pulses received and buffered at 24ppq

h4yn0nnym0u5e · Nov 5, 2023

Might this be caused by setting up all your handlers in setup(), and there being time for MIDI clock messages to arrive between then and your first calls to usbMIDI.read() in loop()?

If so, you could try moving the call to setupUsbClientMidi() into your if (firstloop) {} block, after the while loop. Then that usbMIDI.read() will temporarily discard messages it doesn’t handle, as you expect…

snowsh · Nov 5, 2023

h4yn0nnym0u5e said:
Might this be caused by setting up all your handlers in setup(), and there being time for MIDI clock messages to arrive between then and your first calls to usbMIDI.read() in loop()?

If so, you could try moving the call to setupUsbClientMidi() into your if (firstloop) {} block, after the while loop. Then that usbMIDI.read() will temporarily discard messages it doesn’t handle, as you expect…

I have tried this. No good. I suspect this is not due to attaching the interrupt handlers, instead its happening as soon as the midi library is included.

Now that i think about it, I am beginning to doubt the midi.read() ever worked, because i had to manually reattach the t4.1 as a midi device in the DAW (reaper) every time i restarted my teensy.

To put this into a bit more context, my project is a live sequencer of sorts. in the event of a restart during a live event, this rush of clock events is not acceptable.

I'm not sure if this is the correct library that is being used, I have had a hard time finding it to look at the code:
https://github.com/lathoub/Arduino-USBMIDI/blob/master/src/USB-MIDI_defs.h

if this is the correct library, I see that there is a separation between regular midi events like note on/off, at, control change, program/bank - and then there is the sysex, clock and others. I wonder if the midi.read() is only relevant to the first list of items? (Sorry for my poor explanation)

Code:

static uint8_t type2cin[][2] = { {MidiType::InvalidType,0}, {MidiType::NoteOff,8}, {MidiType::NoteOn,9}, {MidiType::AfterTouchPoly,0xA}, {MidiType::ControlChange,0xB}, {MidiType::ProgramChange,0xC}, {MidiType::AfterTouchChannel,0xD}, {MidiType::PitchBend,0xE} };

static uint8_t system2cin[][2] = { {MidiType::SystemExclusive,0}, {MidiType::TimeCodeQuarterFrame,2}, {MidiType::SongPosition,3}, {MidiType::SongSelect,2}, {0,0}, {0,0}, {MidiType::TuneRequest,5}, {MidiType::SystemExclusiveEnd,0}, {MidiType::Clock,0xF}, {0,0}, {MidiType::Start,0xF}, {MidiType::Continue,0xF}, {MidiType::Stop,0xF}, {0,0}, {MidiType::ActiveSensing,0xF}, {MidiType::SystemReset,0xF} };

PaulS · Nov 5, 2023

So you are not using the Teensyduino supplied MIDI library: https://www.pjrc.com/teensy/td_midi.html ?

Paul

snowsh · Nov 5, 2023

yes i am using the teensyduino midi library. Please point me to its GITHUB

PaulS · Nov 5, 2023

Here you go: https://github.com/PaulStoffregen/cores/tree/master/usb_midi

Paul

snowsh · Nov 5, 2023

PaulS said:
Here you go: [url]https://github.com/PaulStoffregen/cores/tree/master/usb_midi[/URL]

Paul

@PaulS Thankyou! I have taken a look over it, but nothing is totally obvious to me. Maybe I should look at this:

Code:

// discard any buffered input
void usb_serial_class::flush()
{
        uint8_t intr_state;

        if (usb_configuration) {
                intr_state = SREG;
                cli();
        UENUM = DEBUG_RX_ENDPOINT;
                while ((UEINTX & (1<<RWAL))) {
                        UEINTX = 0x6B;
                }
                SREG = intr_state;
        }
    prev_byte = 0;
}

// transmit a character.
#if ARDUINO >= 100
size_t usb_serial_class::write(uint8_t c)
#else
#define setWriteError()
void usb_serial_class::write(uint8_t c)
#endif
{
        //static uint8_t previous_timeout=0;
        uint8_t timeout, intr_state;

        // if we're not online (enumerated and configured), error
        if (!usb_configuration) goto error;
        // interrupts are disabled so these functions can be
        // used from the main program or interrupt context,
        // even both in the same program!
        intr_state = SREG;
        cli();
        UENUM = DEBUG_TX_ENDPOINT;
        // if we gave up due to timeout before, don't wait again
#if 0
    // this seems to be causig a lockup... why????
        if (previous_timeout) {
                if (!(UEINTX & (1<<RWAL))) {
                        SREG = intr_state;
                        return;
                }
                previous_timeout = 0;
        }
#endif
        // wait for the FIFO to be ready to accept data
        timeout = UDFNUML + TRANSMIT_TIMEOUT;
        while (1) {
                // are we ready to transmit?
                if (UEINTX & (1<<RWAL)) break;
                SREG = intr_state;
                // have we waited too long?  This happens if the user
                // is not running an application that is listening
                if (UDFNUML == timeout) {
                        //previous_timeout = 1;
                        goto error;
                }
                // has the USB gone offline?
                if (!usb_configuration) goto error;
                // get ready to try checking again
                intr_state = SREG;
                cli();
                UENUM = DEBUG_TX_ENDPOINT;
        }
        // actually write the byte into the FIFO
        UEDATX = c;
        // if this completed a packet, transmit it now!
        if (!(UEINTX & (1<<RWAL))) {
        UEINTX = 0x3A;
            debug_flush_timer = 0;
    } else {
            debug_flush_timer = TRANSMIT_FLUSH_TIMEOUT;
    }
        SREG = intr_state;
#if ARDUINO >= 100
        return 1;
#endif
error:
#if ARDUINO >= 100
        setWriteError();
        return 0;
#else
    return;
#endif
}

However i thought that flush() was to do with TX midi not RX. Maybe the comment "// discard any buffered input" is misleading.

OK, so I am doing some more debug to try to track this.... Note my DAW is playing and sending a midi clock to the device in all scenarios:

I have added this to the end of setup():

Code:

  Serial.println(F("usbMIDI.read()"));

  int readCounter = 0;
  while (int p = usbMIDI.read())
  {
    Serial.print(F("readCounter "));
    Serial.print(readCounter);
    Serial.print(F(": "));
   
    Serial.println(p);
   
    readCounter++;
   
  }

with my call to setupUsbClientMidi() towards the start of setup() here is the debug result:

usbMIDI.read()
readCounter 0: 1
readCounter 1: 1
readCounter 2: 1
readCounter 3: 1
readCounter 4: 1
readCounter 5: 1
readCounter 6: 1
readCounter 7: 1

the device gets to loop and races to around 4 measures, 4 beats.

with my call to setupUsbClientMidi() at the end of setup() here is the debug result:

setup midi stuff - setup midi stuff done
usbMIDI.read()
readCounter 0: 1
readCounter 1: 1
readCounter 2: 1
readCounter 3: 1
readCounter 4: 1
readCounter 5: 1
readCounter 6: 1
readCounter 7: 1

Exactly the same.... the device gets to loop and races to around 4 measures, 4 beats. It makes no difference where these interrupt handlers are declared......

Now with no declaration of setupUsbClientMidi() - and as I expect the device does not respond to midi clock once in loop:

usbMIDI.read()
readCounter 0: 1
readCounter 1: 1
readCounter 2: 1
readCounter 3: 1
readCounter 4: 1
readCounter 5: 1

interesting. The only midi to my knowledge being sent from the DAW is the clock. I put the missing 2 reads down to the time spend running setupUsbClientMidi(), but I am not even certain of this. The whole of setup takes about 20 seconds. There are structs to initiate and data to be loaded from SD. I doubt very much this function takes ~5 seconds (20/8)*2 and just 8 records would hardly be representative of a midi clock running at 24ppq for 20 seconds!

Code:

void setupUsbClientMidi()
{
  Serial.print(F("setup midi stuff"));


  usbMIDI.setHandleRealTimeSystem(RealTimeSystem);


  usbMIDI.setHandleSongPosition(mySongPosition);                          // used to set a start point based on where the daw is set at start

  usbMIDI.setHandleNoteOn(myUsbClientNoteOn);   
  usbMIDI.setHandleNoteOff(myUsbClientNoteOff);


  usbMIDI.setHandleSystemExclusive(mySystemExclusiveChunk);

 
  Serial.println(F(" - setup midi stuff done"));
}

So this leaves me thinking that usbMIDI.read() is not accessing the midi clock events. Perhaps its hitting active sensing sent from reaper? But then according to the PJRC documentation re active sensing, these should be sent every 300ms! So probably not that.......
Stumped. Right now I am considering writing a special clock function to run at the start of the first loop just to flush out these buffered clock messages, but I don't want to botch in a hacky workaround

Any more ideas? If needed I will put together a sketch to reproduce the behavior, unless maybe @PaulStoffregen has more ideas?

snowsh · Nov 5, 2023

in an effort to determine what these events are that are caught with usbMIDI.read(), I have added this:

Code:

//in setup:
  Serial.println(F("usbMIDI.read()"));

  int readCounter = 0;
  while (int p = usbMIDI.read())
  {
    Serial.print(F("-----readCounter "));
    Serial.print(readCounter);
    Serial.print(F(": "));
    
    Serial.println(p);
    
    readCounter++;

    debugIncomingMIDI();
    
  }


// from example: teeny-usb midi - inputRead.ino: (renamed)

void debugIncomingMIDI(void) {
  byte type, channel, data1, data2, cable;

  // fetch the MIDI message, defined by these 5 numbers (except SysEX)
  //
  type = usbMIDI.getType();       // which MIDI message, 128-255
  channel = usbMIDI.getChannel(); // which MIDI channel, 1-16
  data1 = usbMIDI.getData1();     // first data byte of message, 0-127
  data2 = usbMIDI.getData2();     // second data byte of message, 0-127
  cable = usbMIDI.getCable();     // which virtual cable with MIDIx8, 0-7

  // uncomment if using multiple virtual cables
  //Serial.print("cable ");
  //Serial.print(cable, DEC);
  //Serial.print(": ");

  // print info about the message
  //
  switch (type) {
    case usbMIDI.NoteOff: // 0x80
      Serial.print("Note Off, ch=");
      Serial.print(channel, DEC);
      Serial.print(", note=");
      Serial.print(data1, DEC);
      Serial.print(", velocity=");
      Serial.println(data2, DEC);
      break;

    case usbMIDI.NoteOn: // 0x90
      Serial.print("Note On, ch=");
      Serial.print(channel, DEC);
      Serial.print(", note=");
      Serial.print(data1, DEC);
      Serial.print(", velocity=");
      Serial.println(data2, DEC);
      break;

    case usbMIDI.AfterTouchPoly: // 0xA0
      Serial.print("AfterTouch Change, ch=");
      Serial.print(channel, DEC);
      Serial.print(", note=");
      Serial.print(data1, DEC);
      Serial.print(", velocity=");
      Serial.println(data2, DEC);
      break;

    case usbMIDI.ControlChange: // 0xB0
      Serial.print("Control Change, ch=");
      Serial.print(channel, DEC);
      Serial.print(", control=");
      Serial.print(data1, DEC);
      Serial.print(", value=");
      Serial.println(data2, DEC);
      break;

    case usbMIDI.ProgramChange: // 0xC0
      Serial.print("Program Change, ch=");
      Serial.print(channel, DEC);
      Serial.print(", program=");
      Serial.println(data1, DEC);
      break;

    case usbMIDI.AfterTouchChannel: // 0xD0
      Serial.print("After Touch, ch=");
      Serial.print(channel, DEC);
      Serial.print(", pressure=");
      Serial.println(data1, DEC);
      break;

    case usbMIDI.PitchBend: // 0xE0
      Serial.print("Pitch Change, ch=");
      Serial.print(channel, DEC);
      Serial.print(", pitch=");
      Serial.println(data1 + data2 * 128, DEC);
      break;

    case usbMIDI.SystemExclusive: // 0xF0
      // Messages larger than usbMIDI's internal buffer are truncated.
      // To receive large messages, you *must* use the 3-input function
      // handler.  See InputFunctionsComplete for details.
      Serial.print("SysEx Message: ");
      printBytes(usbMIDI.getSysExArray(), data1 + data2 * 256);
      Serial.println();
      break;

    case usbMIDI.TimeCodeQuarterFrame: // 0xF1
      Serial.print("TimeCode, index=");
      Serial.print(data1 >> 4, DEC);
      Serial.print(", digit=");
      Serial.println(data1 & 15, DEC);
      break;

    case usbMIDI.SongPosition: // 0xF2
      Serial.print("Song Position, beat=");
      Serial.println(data1 + data2 * 128);
      break;

    case usbMIDI.SongSelect: // 0xF3
      Serial.print("Sond Select, song=");
      Serial.println(data1, DEC);
      break;

    case usbMIDI.TuneRequest: // 0xF6
      Serial.println("Tune Request");
      break;

    case usbMIDI.Clock: // 0xF8
      Serial.println("Clock");
      break;

    case usbMIDI.Start: // 0xFA
      Serial.println("Start");
      break;

    case usbMIDI.Continue: // 0xFB
      Serial.println("Continue");
      break;

    case usbMIDI.Stop: // 0xFC
      Serial.println("Stop");
      break;

    case usbMIDI.ActiveSensing: // 0xFE
      Serial.println("Actvice Sensing");
      break;

    case usbMIDI.SystemReset: // 0xFF
      Serial.println("System Reset");
      break;

    default:
      Serial.println("Opps, an unknown MIDI message type!");
  }
}

and here is the result:

usbMIDI.read()
-----readCounter 0: 1
Clock
-----readCounter 1: 1
Clock
-----readCounter 2: 1
Clock
-----readCounter 3: 1
Clock
-----readCounter 4: 1
Clock
-----readCounter 5: 1
Clock
-----readCounter 6: 1
Clock

its clock! but only 7 records??? and its still racing through 4 measures and 4 beats!

snowsh · Nov 8, 2023

In response to @PaulS - you last post (now missing) just before the server went down and the site rebuild.

You showed me your two test sketches to try to replicate the issue:

My reply:

Thanks for your time here Paul. Yes your last paragraph explains the issue.

My environment:
I have reaper playing. I am using it primarily as a VST host for instruments, and a recording / debug environment. It is connected to my t4.1 over USB. My project is compiled as USB type serial+MIDIx4.

Reaper is usually playing. Any time I upload to my T4.1, or simply restart it, Reaper continues to play. All it is sending is a midi clock. It can of course send any other midi data, but in these scenarios it is just a clock. The reaper project is otherwise empty of any track note data. If reaper is not playing during startup, this issue does not happen. I need to fix this because my project is partly intended to be a live sequencer tool. Any event where the T4.1 looses connection, or needs a restart this rush of notes would be unacceptable in a live performance. To give you an idea of what this is all about, take a look here:

- this video does not show the rush problem, but gives an idea of the device.

So as I develop some code and want to test, I hit upload. During compile the T4.1 continues to run and sends its generated midi to reaper - which plays the VST instruments.
When the sketch is compiled, it begins upload to the T4.1, at which point is is no longer running the sketch and not connected to reaper. Any notes that are currently on continue to play as there is no note-off signal.

Once the upload is complete, During the setup() after midi is initiated, it sends a series of all note off CC messages. This works. Remember that reaper is still playing and sending out its clock.

As soon as setup is complete, and we go into loop() there is a rush of notes played in reaper and my display (which is tied into the clock) updates rapidly, and this flurry (rush) of notes are sent to reaper. When the rush has finished I can see my timer on the primary display has reached around 4:4:00. This is not always the same value, but is hard to determine because the presence of the clock forces my T4.1 into sequencer enable mode so in effect it is "playing" and the clock continues. Once the rush has passed, which lasts no more than 1/2 of a second, things settle down and the device plays as expected.

I hope your sketch manages to reproduce the issue.

I need to devise a method to ignore any received midi data during the setup phase. while (usbMIDI.read()) {}; has no effect. The debug catches 8 clock records, but again sometimes this is different. I have seen 6,7,8, and 9 clock records. Either way that is odd to me as 96 clocks per beat (4 quarter notes) would imply that there has been at least 1395 clocks for 4 measures, 4 beats. (4/4 timing)

In your screenshot I see that your setup in the RX T4.1 only runs the while(USBmidi.read()){}; this would give no time for any clock signals to buffer - I would suggest you put something before the while statement to cause around 20 seconds for the setup to complete. I haven't yet done anything to recreate this in a simpler sketch, but it occurred to me that delay(20000); may not give the desired result (I am not sure if delay() would block everything and effect incoming USB or interrupt buffering.) I was going to do this by running some SD file transfers as this is the main thing that is taking the time in my setup routine. Note My SD read is not a single file, I load up about 400 small files. While the project is running, its settings are updated to SD regularly when edits are made, and keeping these files small mean I don't have any lags while playing music.

PaulS · Nov 8, 2023

snowsh said:
In response to @PaulS - you last post (now missing) just before the server went down and the site rebuild.

Yes, found that out too.
Anyway, during the forum outage I installed Reaper, made a simple drum track at 120bpm, and was able to output MIDI clock messages.
I updated the RX code a bit:

C++:

// https://forum.pjrc.com/threads/73836-flush-incoming-midi-buffer-at-end-of-setup
#include <MIDI.h>

unsigned long currentRXtime;
unsigned long previousRXtime;

void setup() {
  while (usbMIDI.read()) {} // ignore incoming messages
}

void loop() {
  if (usbMIDI.read()) {
    if (usbMIDI.Clock) {
      currentRXtime = micros();
      Serial.print(currentRXtime - previousRXtime); Serial.print("\t");Serial.println("F8");
      previousRXtime = currentRXtime;
    }
  }
}

Have some observations but will elaborate on those in a future message - have to go to work now.

Paul

PaulS · Nov 9, 2023

Allright, the one important observation I had.
Steps to reproduce:
1. Reaper (V7.02) plays track and outputs MIDI clock message 'F8' continuously
2. Teensy receives these F8 messages and shows them on the serial monitor
3. Disconnect Teensy from USB (or upload new code)
4. Reconnect Teensy to USB (or rebooting after upload)
5. No F8 messages received from Reaper anymore
6. Have to 'Reset all MIDI devices' in Reaper to resume sending out F8 messages
7. Teensy receives these F8 messages and shows them on the serial monitor

Do you observe this too or does reconnecting happen automatically?

Paul

snowsh · Nov 9, 2023

No it reconnects automatically. Reaper V6.82/win64 rev 07910d.

C++:

void setupUsbClientMidi()
{
  Serial.print(F("setup midi stuff"));
  usbMIDI.setHandleRealTimeSystem(RealTimeSystem);
  usbMIDI.setHandleSongPosition(mySongPosition);                          // used to set a start point based on where the daw is set at start
  usbMIDI.setHandleNoteOn(myUsbClientNoteOn);   
  usbMIDI.setHandleNoteOff(myUsbClientNoteOff);
  usbMIDI.setHandleSystemExclusive(mySystemExclusiveChunk);
  Serial.println(F(" - setup midi stuff done"));
}

Note I am not using midi.read(). I have all event setup with interrupt handlers:
I am also using active sensing in loop:

C++:

//called in loop:

  usbMidiActiveSensingHandler();
void usbMidiActiveSensingHandler()                                // might proove useful
{
  if (enableActiveSensing)
  {
    if ((currentMillis - lastUsbMidiActiveSensingRefreshTime) > usbMidiActiveSensingDelay)
    {
      //Serial.println(F("usb midi active sensing"));

      lastUsbMidiActiveSensingRefreshTime = currentMillis;

      usbMidiActiveSensing();
    }
  }
}

void usbMidiActiveSensing()
{
  usbMIDI.sendRealTime(usbMIDI.ActiveSensing);
  // TODO: add MIDI and MIDI2

}

My RealTimeSystem function is in two parts (overloading) - I use a 2nd argument to determine if I am reading external or internal midi clock.

C++:

void RealTimeSystem(byte realtimebyte)                                                            // overloading - this function deals with passing the incoming USB midi clock on.........
{
  if (config.midiClockType == MIDI_CLOCK_INTERNAL && realtimebyte == CLOCK)                       // if we are set to internal clock, check for incoming clock signals here and ignore
  {
    //Serial.println(F("ignoring external clock. internal clock mode"));
  }
  else                                                                                            // allow other realtime commands through.
  {
    RealTimeSystem(realtimebyte, false);
  }
}

void RealTimeSystem(byte realtimebyte, bool isInternal)                                           // overloading - main function that handles all our incoming midi and the clock. Arg 2 isInternal determines the clock source
{
    switch (realtimebyte)
    {
      case CLOCK:
        if (sequencerEnable)
        {
          sendMIDIclockOut();

          if ((config.midiClockType == MIDI_CLOCK_INTERNAL && isInternal) || (config.midiClockType == MIDI_CLOCK_EXTERNAL && !isInternal))    // sort the clock source internal vs external
          {
            runRealTimeClock(isInternal);
          }
        }
        else                                                                                        // we are recieving a clock, so we need to enable the sequencer etc....
        {
          sequencerEnable = true;
          isPaused = false;

          resetCounters();                                                                          // crash in here

          if ((config.midiClockType == MIDI_CLOCK_INTERNAL && isInternal) || (config.midiClockType == MIDI_CLOCK_EXTERNAL && !isInternal))
          {
            if (clockDebug) Serial.println("runRealTimeClock(bool isInternal)");
            runRealTimeClock(isInternal);
            if (clockDebug) Serial.println("runRealTimeClock(bool isInternal) done");
          }
        }
        break;
      case START:
        setInternalClockMicros();
        isPaused = false;
        sequencerEnable = true;

        if (isInternal)
        {
          internalClock.begin(isrInternalClock, internalClockRateMicros); //(60000 / (bpm/ppqn)) );
        }

        runRealTimeClock(isInternal);
        break;
      case CONTINUE:                                                                                // start continue
        setInternalClockMicros();
        isPaused = false;
        sequencerEnable = true;

        if (isInternal)
        {
          Serial.println(F("if(isInternal) CONTINUE\n uSec: "));
          Serial.println(internalClockRateMicros);

          internalClock.begin(isrInternalClock, internalClockRateMicros); //(60000 / (bpm/ppqn)) );
        }

        runRealTimeClock(isInternal);
        break;
      case STOP:
        setInternalClockMicros();
        allActiveNotesOff();
        sequencerEnable = false;
        if (isInternal)
        {
          Serial.println(F("if(isInternal) STOP"));
          internalClock.end();
        }
        allNotesOff();
        break;
  }
}

I must stress that it is important to make setup do something to take up around 20 seconds - this way you will hopefully reproduce the bug. I have a routine where i can load a new project from SD. This is called from the UI so is within loop. This sets sequencerEnable to false, loads the project files from SD - this takes around 10 seconds and the same "rush" effect is seen.

Ignore internal midi clock.

snowsh · Nov 9, 2023

A further observation. You can manually reconnect reaper during the setup(). This will hopefully mean the teensy is receiving the clocks during setup() and you will observe the bug.

joepasquariello · Nov 9, 2023

20 seconds is a very long time. Can you wait to create your Midi object after you do all of this file I/O?

snowsh · Nov 10, 2023

joepasquariello said:
20 seconds is a very long time. Can you wait to create your Midi object after you do all of this file I/O?

As already tested it makes no difference.

PaulS · Nov 10, 2023

snowsh said:
No it reconnects automatically. Reaper V6.82/win64 rev 07910d.

That's weird. Can't get Reaper to reconnect automatically.
I also found several posts on the internet that it does not work without "Reset all MIDI devices".
Even found this post that someone made a workaround for that issue: https://forum.cockos.com/showpost.php?p=2481951 .
Perhaps the difference between V6.82 and V7.03?

Anyway, I was able to catch the issue you saw!
Here is the code I used:

C++:

#include <MIDI.h>

unsigned long currentRXtime;
unsigned long previousRXtime;

void setup() {
  delay(20000);  // wait 20 secs
}

void loop() {
  if (usbMIDI.read()) {
    if (usbMIDI.Clock) {
      currentRXtime = micros();
      Serial.print(currentRXtime - previousRXtime); Serial.print("\t"); Serial.println("F8");
      previousRXtime = currentRXtime;
    }
  }
}

While Reaper was sending out clocks, I rebooted the Teensy, hit the "Reset all MIDI devices" button in Reaper and waited for 20 secs.
Here is what the serial monitor shows:

20300001 F8

3 F8

2 F8

3 F8

2 F8

2 F8

6 F8

273 F8

248 F8

255 F8

246 F8

254 F8

248 F8

250 F8

138 F8

241 F8

132 F8

239 F8

228 F8

138 F8

151 F8

211 F8

136 F8

124 F8

127 F8

126 F8

126 F8

220 F8

154 F8

126 F8

220 F8

155 F8

125 F8

124 F8

125 F8

124 F8

126 F8

124 F8

126 F8

123 F8

128 F8

122 F8

128 F8

123 F8

125 F8

127 F8

123 F8

127 F8

123 F8

126 F8

126 F8

125 F8

256 F8

118 F8

129 F8

217 F8

155 F8

135 F8

136 F8

109 F8

145 F8

115 F8

119 F8

116 F8

123 F8

130 F8

121 F8

126 F8

123 F8

127 F8

124 F8

124 F8

126 F8

124 F8

127 F8

125 F8

123 F8

124 F8

126 F8

126 F8

125 F8

128 F8

120 F8

128 F8

122 F8

126 F8

125 F8

125 F8

125 F8

125 F8

129 F8

123 F8

122 F8

128 F8

123 F8

127 F8

123 F8

128 F8

123 F8

125 F8

126 F8

126 F8

121 F8

128 F8

124 F8

128 F8

122 F8

126 F8

124 F8

124 F8

126 F8

124 F8

126 F8

126 F8

125 F8

123 F8

128 F8

122 F8

127 F8

124 F8

128 F8

121 F8

126 F8

124 F8

125 F8

126 F8

125 F8

127 F8

124 F8

128 F8

119 F8

127 F8

125 F8

124 F8

143 F8

111 F8

142 F8

111 F8

119 F8

123 F8

126 F8

133 F8

118 F8

126 F8

123 F8

130 F8

121 F8

223 F8

151 F8

127 F8

128 F8

121 F8

123 F8

127 F8

123 F8

126 F8

126 F8

124 F8

126 F8

124 F8

129 F8

122 F8

124 F8

127 F8

123 F8

129 F8

123 F8

123 F8

126 F8

221 F8

154 F8

223 F8

151 F8

233 F8

144 F8

133 F8

142 F8

111 F8

138 F8

112 F8

138 F8

107 F8

121 F8

121 F8

125 F8

125 F8

125 F8

127 F8

129 F8

255 F8

114 F8

127 F8

123 F8

127 F8

123 F8

129 F8

120 F8

127 F8

126 F8

122 F8

127 F8

123 F8

128 F8

123 F8

129 F8

121 F8

127 F8

124 F8

125 F8

126 F8

125 F8

122 F8

128 F8

123 F8

127 F8

122 F8

125 F8

128 F8

123 F8

125 F8

129 F8

123 F8

123 F8

127 F8

123 F8

129 F8

123 F8

123 F8

125 F8

127 F8

122 F8

125 F8

128 F8

123 F8

126 F8

124 F8

129 F8

121 F8

125 F8

127 F8

125 F8

122 F8

128 F8

124 F8

125 F8

124 F8

126 F8

124 F8

126 F8

124 F8

127 F8

122 F8

130 F8

124 F8

219 F8

156 F8

124 F8

124 F8

126 F8

123 F8

127 F8

123 F8

127 F8

122 F8

127 F8

123 F8

128 F8

124 F8

124 F8

127 F8

126 F8

121 F8

125 F8

127 F8

124 F8

126 F8

124 F8

126 F8

126 F8

125 F8

122 F8

128 F8

124 F8

125 F8

126 F8

125 F8

123 F8

127 F8

123 F8

127 F8

122 F8

136 F8

115 F8

127 F8

123 F8

125 F8

127 F8

126 F8

121 F8

128 F8

125 F8

125 F8

123 F8

128 F8

121 F8

125 F8

128 F8

123 F8

127 F8

123 F8

231 F8

146 F8

220 F8

157 F8

139 F8

113 F8

148 F8

112 F8

122 F8

111 F8

125 F8

128 F8

123 F8

161 F8

101 F8

115 F8

122 F8

126 F8

126 F8

126 F8

127 F8

122 F8

124 F8

126 F8

124 F8

127 F8

124 F8

126 F8

122 F8

127 F8

125 F8

123 F8

127 F8

125 F8

124 F8

132 F8

120 F8

122 F8

126 F8

126 F8

123 F8

128 F8

123 F8

126 F8

124 F8

127 F8

123 F8

127 F8

122 F8

126 F8

131 F8

123 F8

120 F8

127 F8

126 F8

122 F8

126 F8

127 F8

123 F8

124 F8

128 F8

124 F8

125 F8

125 F8

125 F8

127 F8

123 F8

126 F8

124 F8

126 F8

125 F8

126 F8

123 F8

125 F8

124 F8

125 F8

127 F8

123 F8

127 F8

123 F8

129 F8

123 F8

123 F8

128 F8

122 F8

127 F8

123 F8

127 F8

123 F8

125 F8

125 F8

128 F8

121 F8

128 F8

123 F8

127 F8

122 F8

126 F8

127 F8

124 F8

124 F8

127 F8

121 F8

129 F8

124 F8

125 F8

123 F8

128 F8

123 F8

127 F8

126 F8

123 F8

125 F8

124 F8

125 F8

129 F8

121 F8

124 F8

128 F8

123 F8

127 F8

123 F8

126 F8

123 F8

128 F8

126 F8

123 F8

126 F8

123 F8

127 F8

123 F8

230 F8

147 F8

136 F8

118 F8

119 F8

125 F8

125 F8

126 F8

125 F8

127 F8

122 F8

159 F8

103 F8

124 F8

119 F8

120 F8

127 F8

125 F8

126 F8

121 F8

125 F8

130 F8

121 F8

135 F8

124 F8

115 F8

128 F8

122 F8

125 F8

126 F8

126 F8

125 F8

123 F8

124 F8

127 F8

123 F8

127 F8

127 F8

123 F8

129 F8

121 F8

125 F8

127 F8

121 F8

125 F8

126 F8

126 F8

126 F8

122 F8

127 F8

125 F8

123 F8

125 F8

127 F8

130 F8

118 F8

128 F8

126 F8

123 F8

131 F8

119 F8

125 F8

128 F8

124 F8

125 F8

122 F8

127 F8

124 F8

127 F8

228 F8

147 F8

125 F8

124 F8

129 F8

118 F8

127 F8

126 F8

123 F8

130 F8

121 F8

126 F8

124 F8

127 F8

123 F8

129 F8

123 F8

252 F8

218 F8

154 F8

147 F8

109 F8

121 F8

125 F8

122 F8

127 F8

123 F8

127 F8

122 F8

126 F8

130 F8

121 F8

124 F8

125 F8

124 F8

128 F8

122 F8

127 F8

124 F8

129 F8

120 F8

128 F8

123 F8

124 F8

125 F8

128 F8

123 F8

127 F8

123 F8

132 F8

155 F8

123 F8

113 F8

112 F8

117 F8

124 F8

124 F8

128 F8

121 F8

128 F8

161 F8

113 F8

228 F8

144 F8

121 F8

124 F8

124 F8

116 F8

117 F8

128 F8

123 F8

124 F8

124 F8

126 F8

124 F8

128 F8

122 F8

125 F8

125 F8

125 F8

125 F8

126 F8

126 F8

126 F8

124 F8

126 F8

126 F8

123 F8

125 F8

127 F8

124 F8

122 F8

125 F8

125 F8

127 F8

125 F8

124 F8

127 F8

124 F8

124 F8

126 F8

125 F8

127 F8

121 F8

127 F8

128 F8

218 F8

154 F8

23378 F8

19506 F8

19494 F8

26895 F8

21614 F8

19867 F8

19623 F8

19220 F8

19549 F8

19635 F8

24829 F8

23142 F8

20484 F8

19486 F8

19658 F8

19592 F8

19529 F8

19377 F8

27002 F8

21375 F8

19872 F8

20374 F8

19499 F8

20002 F8

19252 F8

19092 F8

27282 F8

21126 F8

20001 F8

Time is in micro-seconds. So the first F8 came in after ~20secs, then a huge rush of F8's with an interval of ~125usec and then finally the steady cadence of F8's with an interval of ~21msec.

Paul

snowsh · Nov 10, 2023

Aha, well reaPack is the reason its reconnecting then. I had installed it recently too but hadn't made the connection that it was responsible for the auto reconnect.
So your first F8 is dumped at 20300001 uS from power on - as the initial declaration of unsigned long previousRXtime must be initializing to 0. the delay(20000) would tally with this. The the rest of the time displays are determined by the time the while loop is processing them. This doesn't really get us an idea of when the clock was actually received - I'm going with reaper being rock steady here and simply pumping out clocks.. I'm convinced there must be some buffering happening, either in the midi library or probably more likely the USB library.

Its very odd that in my attempts to serial.print these clocks, I only get ~ 8 values - but with 4 beats * 4 measures at 24ppq I knew there had to be around 1536 clocks! I'm glad your sketch shows this!

So, we need to find a way to clear these clock events received during setup() phase. It has to be in midi or the USB somewhere, but that is beyond me at the moment.

snowsh · Nov 10, 2023

@PaulS I have updated your test example to illustrate better. You will need to install reaPack. For some reason this example is not 100% auto connecting, but it does sometimes and when it does, you experience the rush of notes sent back to the DAW. My main program connects 100% of the time.

I have found that if you click "reset all midi devices" in reaper preferences as soon as the upload is complete, it will connect. this seems to be confirmed with the reaper preferences window flickering once. Clicking it later during the setup() can cause connection to fail.

I have tried without the active sensing, and it never auto connects so i suspect that is part of the connection process.

Delay adjusted to 10 seconds to save boredom

to hear the drum pattern, you could add free VSTi tromine c78 on cable 0, channel 1. https://www.kvraudio.com/product/tromine-c-by-marvin-vst

C++:

#include <MIDI.h>                                                        

unsigned long currentRXtime;
unsigned long previousRXtime;
uint8_t counter = 0;
unsigned long lastUsbMidiActiveSensingRefreshTime = 0;
unsigned long usbMidiActiveSensingDelay = 300;                          // how often to send a midi active sensing command
bool enableActiveSensing = true;
elapsedMillis currentMillis = 0;
elapsedMicros currentMicros = 0;                                        // greater resolution for internal clock BPM

//-------------------------------------------------------------------------  MIDI byte addresses

const byte CLOCK = 248;

#define PPQ_96 96

//-------------------------------------------------------------------------

void setup()
{
  Serial.begin(57600);
  Serial.println(F("serial ready"));

  setupUsbClientMidi();

  if (enableActiveSensing) usbMidiActiveSensing();

  delay(10000);                                                         // wait 10 secs to allow clocks to buffer
  Serial.println(F("end setup"));

  while (usbMIDI.read()) {};                                            // has no effect to clear the clocks that have buffered
}

//-------------------------------------------------------------------------

void loop()
{
  /*
     this next line I have had in place for a long time and it fixed random rush events experienced during loop().
  */
  while (usbMIDI.read()) {};                                            // fix: https://forum.pjrc.com/threads/28282-How-big-is-the-MIDI-receive-buffer - this fixes random rushes of clocks during loop()

  usbMidiActiveSensingHandler();
}

//-------------------------------------------------------------------------

void RealTimeSystem(byte realtimebyte)
{
  switch (realtimebyte)
  {
    case CLOCK:
      currentRXtime = micros();
      Serial.print(counter); Serial.print("\t"); Serial.print(currentRXtime - previousRXtime); Serial.print("\t"); Serial.println(CLOCK, HEX);
      previousRXtime = currentRXtime;

      processClock();

      counter++;

      if (counter >= PPQ_96) counter = 0;
      break;
  }
}

void processClock()
{
  /*
   *  silly little drum pattern generator to hear the rush
   *  example maps notes to free VSTi tromine c78 on cable 0, channel 1. https://www.kvraudio.com/product/tromine-c-by-marvin-vst
   */
 
  if (counter % 24 == 0)                          // kick on beat
  {
    sendMidiNoteOn(36, 100);
    sendMidiNoteOff(36);
  }
  if (counter % 4 == 0)                           // hat 1/6
  {
    if(random(0, 4) == 1)                         // 1 in 4 probability
    {
      sendMidiNoteOn(44, 110);
      sendMidiNoteOff(44);
    }
  }
  if (counter % 8 == 0)                           // random snare based on 1/3
  {
    if(random(0, 4) == 1)                         // 1 in 4 probability
    {
      sendMidiNoteOn(38, 127);
      sendMidiNoteOff(38);
    }
  }
  if (counter % 12 == 0)                          // random metallic based on 1/2
  {
    if(random(0, 4) == 1)                         // 1 in 4 probability
    {
      sendMidiNoteOn(51, 40);
      sendMidiNoteOff(51);
    }
  }
  if (counter % 6 == 0)                           // random rimshot based on 1/4
  {
    if(random(0, 8) == 1)                         // 1 in 8 probability
    {
      sendMidiNoteOn(13, 127);
      sendMidiNoteOff(13);
    }
  }
}

void usbMidiActiveSensingHandler()               
{
  if (enableActiveSensing)
  {
    if ((currentMillis - lastUsbMidiActiveSensingRefreshTime) > usbMidiActiveSensingDelay)
    {
      lastUsbMidiActiveSensingRefreshTime = currentMillis;

      usbMidiActiveSensing();
    }
  }
}

void sendMidiNoteOn(uint8_t note, uint8_t velocity)
{
  usbMIDI.sendNoteOn(note, velocity, 1, 0);     // send note on over midi channel 1, cable 0
}

void sendMidiNoteOff(uint8_t note)
{
  usbMIDI.sendNoteOff(note, 1, 0);              // send note off over midi channel 1, cable 0
}

void setupUsbClientMidi()
{
  Serial.print(F("setup midi stuff"));
  usbMIDI.setHandleRealTimeSystem(RealTimeSystem);
  Serial.println(F(" - setup midi stuff done"));
}

void usbMidiActiveSensing()
{
  usbMIDI.sendRealTime(usbMIDI.ActiveSensing);
}

PaulS · Nov 11, 2023

snowsh said:
@PaulS I have updated your test example to illustrate better. You will need to install reaPack. For some reason this example is not 100% auto connecting, but it does sometimes and when it does, you experience the rush of notes sent back to the DAW. My main program connects 100% of the time.

It turned out to be even easier. The only thing I had to do was adding reaper_automidireset_x64.dll from reaper_automidireset_win_x64.zip (found here) to the REAPER\UserPlugins directory. Restart Reaper and you're done. Works prefectly.

More on your message later today.

Paul

PaulS · Nov 11, 2023

OK, uploaded your latest sketch from message #23.
Saw this on the serial monitor:

This rush of F8's goes on for roughly 4 x 96 times before the normal 21msec F8 cadence comes in.
Setting enableActiveSensing to false or true did not matter.

Moved the handle usbMIDI.setHandleRealTimeSystem(RealTimeSystem) from void setupUsbClientMidi() to the last line of void setup(), but to no avail.

There must be some intermediate buffering going on somewhere that can not be cleared by while (usbMIDI.read()) {};.

Paul

flush incoming midi buffer at end of setup

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member