What makes this tough is the use of Arduino/Teensy libraries. It's hard to tell if there's some hidden global disabling of interrupts lurking somewhere. Like deep down in SPI, or something like that. I don't know how to confirm or deny if this is happening in my compiled code base. What's the best way to do that? Grep all the source files and headers listed during compilation?
Does TyCommander handle FTDI? Or am I doing something wrong?
- Thread starter clinker8
- Start date
joepasquariello
Well-known member
Yes, searching the libraries is the only way that I know of to find such things. If you provide a list of the libraries you are using, I don't mind helping in the search. If I remember correctly, somewhere in this thread you told us that in some cases your ISR waits for an action to complete. Is that correct? If so, I'd say look at that condition carefully and what might delay it from completing.
There are only two ways that your ISR can run longer than you want. One is if there is a higher priority interrupt, and the other is if your ISR calls something that is taking longer than you intended. Even if some library was disabling interrupts, which is unlikely, that could delay entry to your ISR, but would not cause it to take longer to complete. Please tell us the times you've measured. It's very hard to help when you don't provide details.
I think I found something. I set the interrupt priority after the encoder instantiation. It should be before, if it's working as I think the code does. That assumes that the code does get the priority and at the moment, that's not settled. So I can try that.
"font_DroidSansMono.h". Some of these libs call others.
Yes. If the spindle count exceeds Dprime (set from a thread table) then I call a routine doincrement.
When t1 expires, the timer ISR just clears PUL which is the stepper.
I suppose I can get rid of that printf and use something less time consuming. It was only put in to give me some visibility as to what the timeline was, sync wasn't working properly before the printf, or after I put it in.
I haven't put in your logging the cycle count of callback time yet. Let me do that, and I can report back. My scope stuff was merely an indicator and clearly showed something wasn't proper. I've done dozens of trials and there's gobs of data, which makes it hard to remember and make clear.
Going to run the changes to see if I can raise the priority of EncoderTool. Then add the cycle logging.
EncoderTool, TeensyTimerTool, LittleFS, <XPT2046_Touchscreen.h>, <ILI9341_t3n.h>, <ili9341_t3n_font_Arial.h>, <ili9341_t3n_font_ArialBold.h>,
"font_DroidSansMono.h". Some of these libs call others.
Yes. If the spindle count exceeds Dprime (set from a thread table) then I call a routine doincrement.
C-like:
void doincrement( bool adir ) // sets direction and turns on stepper pulse
{
#if defined(mycrumbs)
CrashReport.breadcrumb(4, 200);
#endif
digitalWriteFast(DIR, adir); // set direction
delayNanoseconds(50); // can improve stepper speed by doing this
digitalWriteFast(PUL, HIGH); // start the stepper pulse
t1.trigger(steplen); // switch off after steplen us (2us) we only get 4us
}
C-like:
void pulseme() // turns off stepper pulse after time out
{
#if defined(mycrumbs)
CrashReport.breadcrumb(4, 100);
#endif
digitalWriteFast(PUL, LOW); // ends the stepper pulse
if (synced) {
synced = 0;
firststeptime = mysec();
//digitalWriteFast(tp1, LOW); // move before printing
HWSerial.printf("First Step at t = %3.10lf sec\n", firststeptime );
}
if (mydir==CCW) steppercount += 1;
else steppercount -= 1;
}It's hard to describe valid times. If it was a simple measurement, I would have done it. But it requires a lot of set up, which is manual, and takes time to physically perform. Lots of button presses, movement of the carriage, setting up logging, etc.
I haven't put in your logging the cycle count of callback time yet. Let me do that, and I can report back. My scope stuff was merely an indicator and clearly showed something wasn't proper. I've done dozens of trials and there's gobs of data, which makes it hard to remember and make clear.
Going to run the changes to see if I can raise the priority of EncoderTool. Then add the cycle logging.
Well foo. TyCommander isn't seeing my 3 boards any more after an update to MacOS 15.6. I did the upgrade last night. Can't debug, as I need connection to at least 2 of them. Made the program changes, but can't get any serial data to display, dag nab it. Even put in @joepasquariello 's cycle count. Moved the nvic set priority command before the encoder instantiation. Behaviorally, the longer than 1.42us interrupts have disappeared, so far. That's good. But I can't get any data out of my machines now, I have no console 
. A step forward, and a couple steps back. Drat, and double drat, been foiled again! Can't really tell if this is truly working on my lathe simulator, but the ISR callback is definitely running about 708 ns as a mean with a max of 1.39us during the time when the stepper is pulsed.
Not clear that I was able to program at all. Definitely frustrated at the moment.
Did a reboot. Seemed to fix the problem. Can see all the boards. ISR callback is 680ns or so, max 1.4us. No big pulses any more. Cycle count min = 0.672 us, max = 1.55 us, which is close, but not the same as the scope. Now to check the time while "logging" and a pass.
min = 0.672 us, max = 159.343us !!! Ouch.
Count (arbitrary at start) when set to zero (synced to index) was -7013. By the time I got to logging, it was -2918 to -2921. By the time I set up a syncing pass it had drifted to -2914 to -2917 (four counts) where it stayed for the sync event. By the time I set up the next threading pass, it drifted to -2906 to -2909. It stayed at that count for the sync event. It appears that some time after the sync, something happens and the count skips. These are the encoder counts when IDX=0. These are logged to PSRAM.
Seems to be 76 lines. That's about 19 revolutions of the spindle, (76 / (4 counts per index)) or about the length of the carriage travel, when the carriage is slowing down. Gives me a clue where to check (I think).
I still don't understand this. I do change Dprime on the fly, (Dprime is a Bresenham variable) but only in the main loop when I'm closing into the stop point. Shouldn't affect my encoder callback at all.
. A step forward, and a couple steps back. Drat, and double drat, been foiled again! Can't really tell if this is truly working on my lathe simulator, but the ISR callback is definitely running about 708 ns as a mean with a max of 1.39us during the time when the stepper is pulsed.Not clear that I was able to program at all. Definitely frustrated at the moment.
Did a reboot. Seemed to fix the problem. Can see all the boards. ISR callback is 680ns or so, max 1.4us. No big pulses any more. Cycle count min = 0.672 us, max = 1.55 us, which is close, but not the same as the scope. Now to check the time while "logging" and a pass.
min = 0.672 us, max = 159.343us !!! Ouch.
Count (arbitrary at start) when set to zero (synced to index) was -7013. By the time I got to logging, it was -2918 to -2921. By the time I set up a syncing pass it had drifted to -2914 to -2917 (four counts) where it stayed for the sync event. By the time I set up the next threading pass, it drifted to -2906 to -2909. It stayed at that count for the sync event. It appears that some time after the sync, something happens and the count skips. These are the encoder counts when IDX=0. These are logged to PSRAM.
Code:
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2909
-2906
-2907
-2908
-2902 #### skipped from -2909 expected to -2902, 7 counts.
-2903
-2904
-2905
-2902
-2903
-2904
-2905
-2902
-2903
-2904
-2905I still don't understand this. I do change Dprime on the fly, (Dprime is a Bresenham variable) but only in the main loop when I'm closing into the stop point. Shouldn't affect my encoder callback at all.
Last edited:
h4yn0nnym0u5e
Well-known member
I've got my test code working (as anticipated, as stupid mistake...) and can now log the priority of the executing callback. Turns out you can set it before or after EncoderTool initialisation, it doesn't matter. @clinker8, do you really mean "instantiation"? That implies you're creating it after your program starts executing, rather than having it defined as a static object...
Looking back, the code you posted in #140 looks disturbing, because you're using
My dump function ended up being very terse, with output suitable for putting into a spreadsheet - I often find that a great way to visualise results, filter for errors, etc.
Darn right you can, and must. Everything with the remotest chance of taking significant time must be rigorously excluded from your ISRs / callbacks.
Looking back, the code you posted in #140 looks disturbing, because you're using
print. Never mind that it's (presumably) to a filesystem in PSRAM - you don't really know what's going on under the hood. - Decide what you want in a log record (encoder value(s), timestamp, error flag etc.)
- Define a
structwith members that can hold one log record - Define a big enough array of those structs - in PSRAM if needed, the speed really won't matter
- Triggered by something, fill the array with records, probably one record per callback
- Only when it's safe to do so, you can use any Serial port, filesystem,
print,printfetc. to dump out the results for inspection
C++:
struct logRecord
{
uint32_t cyccnt, quadPos, encPos, priority;
bool wasError;
};
#define NUM_RECORDS (4096*2)
EXTMEM logRecord logRecords[NUM_RECORDS];
volatile int idx = NUM_RECORDS;
void addToLog(uint32_t cyc,uint32_t enc, bool err)
{
if (idx < NUM_RECORDS)
{
logRecords[idx] = {cyc,myQuad.read(),enc,getNVICpriority(),err};
idx++;
}
}addToLog() is called from either the normal or the error callback, with its err parameter set appropriately.My dump function ended up being very terse, with output suitable for putting into a spreadsheet - I often find that a great way to visualise results, filter for errors, etc.
defragster
Senior Member+
Perhaps TyCommander is denied permissions for "Your Safety" after the update?
Try the IDE SerMon? If it can see one or the other then programming should work, and maybe get SerMon output as well.
Only Windows her - and between TyComm and if that failed I'd try multiple instances of IDE - maybe IDE 1 and another IDE 2 - though IDE 2 may allow unique device on seperate instances?
Thanks. So noted. Kind of lazy on that. Your way is much better. Will go through code and try to scrub it.
No I didn't mean instantiation, my mistake. Sometimes (due to ignorance) use the wrong terms.
Last edited:
Some crud that needed a second reboot? 2nd boot seemed to restore TyCommander again. It's working again.
Edit: My code isn't working right, but TyCommander is functional again.
Last edited:
BriComp
Well-known member
Here is a UTube video showing how to set up and get going with one of those logic analysers,
Thanks. Generally, the issue with stuff like this are drivers. Did a brief look earlier and didn't see a driver for a Mac. But maybe I didn't look hard enough. Will look again, because having the ability to do logic analysis is a good thing.
Although, @h4yn0nnym0u5e pointed out to me the error of my ways, re: print and printf statements. Wouldn't be surprised if things are a little better without those commands in critical places. Still plugging away on that.
joepasquariello
Well-known member
Conversion to text is slower than writing binary, but the T4.1 is so fast, a print here and there is not something to worry about to get the convenience of not having to write a parser to see your results.
h4yn0nnym0u5e
Well-known member
Me too … but it’s way more expensive than the cheapy
Maybe it’s just me …
print() is fairly powerful, so I just assume there’s lots going on I don’t know about, and avoid putting it in anything time-critical, like an ISR. It’s so easy to sidestep the issue and use it solely in foreground code, when you know any timing issues are non-existent, or are at least detectable and recoverable, that I never (well, hardly ever…) risk it.For stuff like this, I would tend to reach for the ‘scope, or the Saleae, very early on. But I already have them - not everyone wants to make that investment.
BriComp
Well-known member
If you watched the video you would see that they have MAC software. The cheapie uses Saleae software.Me too … but it’s way more expensive than the cheapy
Maybe it’s just me …print()is fairly powerful, so I just assume there’s lots going on I don’t know about, and avoid putting it in anything time-critical, like an ISR. It’s so easy to sidestep the issue and use it solely in foreground code, when you know any timing issues are non-existent, or are at least detectable and recoverable, that I never (well, hardly ever…) risk it.
For stuff like this, I would tend to reach for the ‘scope, or the Saleae, very early on. But I already have them - not everyone wants to make that investment.
joepasquariello
Well-known member
Print to Serial in an ISR is a bad idea because it would conflict with any use of Serial at non-interrupt level. I regularly use sprintf() and then copy string to RingBuf in ISRs for data logging.
I was on vacation for a while. (Still am.)
Can you explain the syntax of the EXTMEM line?
Code:
EXTMEM logRecord logRecords[NUM_RECORDS];You dump the data in the main loop with some sort of command?
Last edited:
Where do you do the sprintf function? In the ISR? Then copy the resultant string to RingBuf? Then in the main loop after the excitement is over, you spool it out?
h4yn0nnym0u5e
Well-known member
Exactly. The
EXTMEM at the start puts the 8192-element array in PSRAM. That’s probably not necessary, each struct only needs 9 bytes, though for speed the compiler probably makes it 12, so as written it’s only 96kB. But it’s also only big enough for two shaft rotations; make it 4096*20 for 20 rotations, and it’s 960kB and definitely needs to be in PSRAM. And of course there might be other things you want to add to the struct.I would tend to avoid the use of complex functions like
sprintf in an ISR, but if it’s working for @joepasquariello then maybe it’ll work for you. But you will end up with even more data stored, of probably unknown size, and ideally you daren’t dump any of it until after the cutting is done. Then if counts get lost during the dump, you can re-sync before starting the next cut.One of the things that’s undocumented, and thus makes me averse to
sprintf, is that I don’t know if it allocates heap using malloc. That’s has the potential to cause all manner of obscure issues…
h4yn0nnym0u5e
Well-known member
I just took a look at the RingBuf source code (I haven't used it myself, so I could be reaching incorrect conclusions).
The buffer is stored in Teensy's on-chip RAM, so can't be huge. And it relies on writing to a filesystem to empty the buffer in a timely manner; I'd be wary of that, I know for sure SD cards can lock things up for many milliseconds. Maybe a LittleFS_RAM setup in PSRAM would work, and then copy the file to SD or send to Serial after each cut.
The buffer is stored in Teensy's on-chip RAM, so can't be huge. And it relies on writing to a filesystem to empty the buffer in a timely manner; I'd be wary of that, I know for sure SD cards can lock things up for many milliseconds. Maybe a LittleFS_RAM setup in PSRAM would work, and then copy the file to SD or send to Serial after each cut.
I have 8MB of PSRAM. I'll rerun my calculations on how much I can save. Two rotations is not enough time to log, probably need the equivalent of maybe 10-12 seconds, maybe more. That's like 80 revolutions, or 327680 interrupts (@ 400 RPM). So I need to save at most 24 bytes per edge, or I'll fill the PSRAM in 12 seconds. I'll probably half the data.
joepasquariello
Well-known member
A RingBuf can be located in PSRAM via the EXTMEM keyword, just like any other static variable, but since @clinker8 intends to log data while cutting and extract/print later, he doesn't need RingBuf at all. He can just write directly to PSRAM while cutting, and extract/print later.
RingBuf is designed to let you decouple your logging from writing to SD, like this:
[ISR]---->[RingBuf]---->[loop]---->[SD]
If you look at the SdFat example TeensySdioLogger, it shows how to avoid blocking in loop() when the SD is busy, and that's the technique I use. The longest busy times I have seen for the built-in SD with a 32 GB SanDisk Extreme card are about 40 ms, so my rule of thumb is to size the RingBuf for 50 ms of data, and that has always worked. For a data rate of 1 MB/s, the buffer is only 50 KB.
Even though RingBuf is designed for use with SdFat, you can use it as a general-purpose ring buffer, writing to the tail and reading from the head, though there are probably better alternatives if you're not writing to a file.
h4yn0nnym0u5e
Well-known member
Back here you said
I‘m not sure (we’ve never seen much real code) how tolerant the system is to loop execution, and the possibility of it blocking interrupts, but anything that avoids finding out the hard way has to be the best option! So yes, filling PSRAM during a cut and dumping it before the next one seems the simplest option. I’m not sure you learn much more by doing a long cut, so maybe the test cut can be significantly less than 80 turns?
but now
Still, I’m sure you can figure out the right balance…
I‘m not sure (we’ve never seen much real code) how tolerant the system is to loop execution, and the possibility of it blocking interrupts, but anything that avoids finding out the hard way has to be the best option! So yes, filling PSRAM during a cut and dumping it before the next one seems the simplest option. I’m not sure you learn much more by doing a long cut, so maybe the test cut can be significantly less than 80 turns?
It's hard to control computers, press buttons and actuate lathes at the same time. If I'm out of sequence I don't collect the data, or the buffer fills, or machinery breaks, the work piece is ruined or I get injured. None of those things are really appealing, especially the latter items. While the cut is happening, I need to be next to the emergency shut off switch for the lathe, to avoid physical damage. In 200ms a lot can be twisted and broken.Back here you said
but now
Still, I’m sure you can figure out the right balance…
I‘m not sure (we’ve never seen much real code) how tolerant the system is to loop execution, and the possibility of it blocking interrupts, but anything that avoids finding out the hard way has to be the best option! So yes, filling PSRAM during a cut and dumping it before the next one seems the simplest option. I’m not sure you learn much more by doing a long cut, so maybe the test cut can be significantly less than 80 turns?
As for why more turns, the cut is started about at least 10mm before the work piece, the carriage advances and comes to a precise stop before the work piece. Upon sync with the spindle, it then proceeds with the cut. All of this time converts into a bunch more turns. Especially at 400 RPM. If I blink, it is another turn.
I'd like to log that whole sequence, to validate my system level assumptions. It's a lot of data. I may have to add more PSRAM. If reality is different than my system concept, I will need to change my system.
Re: post #197. Actually, I was not being inconsistent. I don't need many rotations (threads) after the first contact. Just enough to establish phase information. What I do need to capture is what is happening before contact AND the few threads after cutting. At the moment, the start logging process is manual, and there's seconds of time of setup prior to carriage movement. Perhaps I can have the logging start upon the button press (if enabled). Even so, there can be seconds of delay, due to me having to actual a manual lever on my lathe. The lever engages the carriage to the lead screw, via the half nuts. (The half nuts physically are in contact, or are not, with the lead screw, depending on lever position.)
Back on this. Trying to catch up with things, and what needs to be done.
1. Fix logging, get rid of print/printf statements
2. Add timer ISR
3. Add HW encoder
4. Hope the preliminary results show enough promise to encourage further debug and refinement
Back on this. Trying to catch up with things, and what needs to be done.
1. Fix logging, get rid of print/printf statements
2. Add timer ISR
3. Add HW encoder
4. Hope the preliminary results show enough promise to encourage further debug and refinement
joepasquariello
Well-known member
Start the logging whenever, at PSRAM address 0, and when you get to the end of PSRAM, jump back to the beginning and keep going. Whenever you STOP logging, you will have the N most recent seconds of data. Would that contain all the data you want?
FYI, I’m working on updates to the QuadEncoder library to support interrupt callbacks and set interrupt priority. I’ve got a working test program that uses the Position Compare feature to interrupt on every edge. It ran overnight last night with 1024 PPR encoder running at 1200 rpm and never missed an edge.
FYI, I’m working on updates to the QuadEncoder library to support interrupt callbacks and set interrupt priority. I’ve got a working test program that uses the Position Compare feature to interrupt on every edge. It ran overnight last night with 1024 PPR encoder running at 1200 rpm and never missed an edge.