Teensy 3.5 sd logging HELP!

Status
Not open for further replies.

BHAYES

Active member
I'm using the Teensy 3.5 with a very popular project called Speeduino. Works very well. I wanted to implement the sd card slot for data logging the real time variables while the engine is running. I've got the sd card slot working and writing. But when it writes to the sd card all the other functions that are running, tach input, coil outputs, injectors and so on, flicker as if they are being interrupted. I'm not the best programmer in the world. That's why I chose the Arduino platform for this. But something is just not working right. I can post the code if needed. But there is a lot of it. LOL.
 
The easy answer for me is to suggest to move all the polling in the 1000+ lines of loop() into timed interrupts and keep only disk operation in the loop().
As I have no understanding what this speeduino requires as timing, I cannot say more.
 
SD cards contain their own micro controller, so have a tendancy to halt writing while they shuffle stuff around to wear balance. So you always have to design for semi random latency in your SD card reads. as WMXZ says the right answer is to have all other timing critical functions interrupt capable and write logs to a RAM buffer and leave a low prority SD card write function to dump data from the buffer to SD card and let the other functions interupt it as needed.

You may be able to get close enough without refactoring all the code by doing things like:
Looking at the logging process and makeing sure you write in efficiently sized blocks
Making sure no overhead with the SD card is happening (reading files system etc)
Make sure you are using the Teensyduino SD card library if you are using the internal SD card
Check if the SD card library is blocking interrupts, and if it will work (slowly) without them

Ugly hack option is to add another micro controller and keep your primary doing all the real time stuff that gets expensive if it hangs up and move all the less critical stuff off to the other CPU where a slight pause does not impact anything mechanical. Primary streams system state via serial, and takes a fixed format input message packet for any user input/adjustment that is smaller than the hardware buffer.
 
Ugly hack option is to add another micro controller and keep your primary doing all the real time stuff that gets expensive if it hangs up and move all the less critical stuff off to the other CPU where a slight pause does not impact anything mechanical. Primary streams system state via serial, and takes a fixed format input message packet for any user input/adjustment that is smaller than the hardware buffer.

Well, I would not call this Ugly, but Smart (Divide and Conquer). One thing I learned in RT DAQ, never-ever mix data acquisition with display and logging. Only if multiple CPU architecture gets too complicated or too slow use SW.
[Rant]I know someone some time ago said on this forum, why using HW if you can do it in SW. I would replay, why should SW always solve poor or faulty HW design? [\Rant]
 
Definately not Ugly. Standard practice in Motorsport and Automotive industry - one controller for engine, one for gearbox, one for dash/instruments, one for logging.

All core ECU functions are timing-critical, an SD card can occasionally easily take 100ms to write. ECU's use non-volatile Flash for internal logging, and cables or bluetooth for log download.

You do not need 32GB. My £1000 off-the-shelf ECU has 2Mb which is enough to debug short periods at high speed, or long periods at slower speeds. I would look for the fastest largest serial FRAM and write little and often

You could implement a 'copy to SD' at initial start-up or even better during a controlled power-down, if you want the flexibilty that removable storage gives.

Best solution is for 'debug' logs to flash with critial high speed error flags (ie engine speed, load, Crank sync, cam sync) and stream out 'normal' logs on RS232 or Can Bus. You can then log the RS232 or Can bus with a dedicated home-made or Industry standard off-the-shelf logger.
 
Had a quick glance at Speeduino Project.
Looks like you have Canbus already implemented.
I would use the teensy 3.6, (ignoring 5v tolerance issues that might need to be addressed) as it has dual Can. Then build a datalogger with another T3.6. You can then allocate a big buffer to most of the RAM and log at ~14000 CanBus messages a second (7000 from each bus if @ 1Mbaud) without having to worry about a misfire caused by a bad SD card.
 
Thanks guys for the input. I liked the fact of having the SD card on the board for this project. Just to reduce the amount of boxes. I do these for motorcycles so I would like to have the smallest package I can get. I think writing to a buffer and then copying to the sd is my best bet for this. Just not sure how to go about it. Any ideas for this?
 
It's usually wise to write blocks of 512 at a time, some libraries for SD have 2 buffers, when one is writing, the writes continue on 2nd buffer while the first is written to SD
You could handle this with 2 separate buffers, or a multidimensional array as well
uint8_t buffer[2][512] as an example

If the writes are causing issues you could offload the data to a second microcontroller which will do the writing to SD while the main one does processing
 
Thanks guys for the input. I liked the fact of having the SD card on the board for this project. Just to reduce the amount of boxes. I do these for motorcycles so I would like to have the smallest package I can get. I think writing to a buffer and then copying to the sd is my best bet for this. Just not sure how to go about it. Any ideas for this?

First step is get ALL existing code out of loop()
after that there are a variety of examples available that could help, search for logging.
To get existing code into Acquisition interrupt level, one would need to know a little bit about real-time constraints, number of sensors, sampling rates, max delays etc.
So the best is you explain us the expected functionality of this code in more detail.
 
Ok. Let's see if I can explain this. Please bare with me. So the real time variables are updated constantly already. So putting them in a buffer is no prob. I want a max of 20 variables each sample. And I only need 10 to 20 samples per second. 10 will work. 20 would be great! Most variables are 1 byte each and some are 2 bytes. I have plenty of onboard memory left if we need to write to memory and the copy to the card later. Only need enough storage for say 10 to 20 seconds of data.
 
Ok. Let's see if I can explain this. Please bare with me. So the real time variables are updated constantly already. So putting them in a buffer is no prob. I want a max of 20 variables each sample. And I only need 10 to 20 samples per second. 10 will work. 20 would be great! Most variables are 1 byte each and some are 2 bytes. I have plenty of onboard memory left if we need to write to memory and the copy to the card later. Only need enough storage for say 10 to 20 seconds of data.

OK,
let us walk through the issue.
so you collect data (say 20 Bytes/second) you wanted to save on uSD.

As UhClem said in another thread https://forum.pjrc.com/threads/5236...n-(Teensy-3-6)?p=179909&viewfull=1#post179909
every access to uSD will be in the order of 2 ms but can be up to 10 to 20 ms (I have measured also over 100 ms once a minute or so, but this depends on uSD quality).

If you collect 512 bytes then you will have this uSD write delay every 25 seconds.

(BTW, I assume you know that all access to uSD are in chunks of 512 Bytes. If you write only 1 Byte, the file system has to read the old 512 Block, add the new Byte and write the 512 Byte block back to uSD)

If your real time requirements can tollerate the expected and possible delays, you have not to worry. If there are mission critical constraints, than you have to elevate these real-world interactions to high priority interrupt levels.
 
20 variables - 20 times per second - is 400 bytes/sec and some are 2 bytes so maybe 512 bytes to write per second or so?

Is this running for a long period or just a minute or two under test?
 
20 variables - 20 times per second - is 400 bytes/sec and some are 2 bytes so maybe 512 bytes to write per second or so?

Is this running for a long period or just a minute or two under test?

I just need 10 to 20 seconds as of right now.
 
OK,
let us walk through the issue.
so you collect data (say 20 Bytes/second) you wanted to save on uSD.

As UhClem said in another thread https://forum.pjrc.com/threads/5236...n-(Teensy-3-6)?p=179909&viewfull=1#post179909
every access to uSD will be in the order of 2 ms but can be up to 10 to 20 ms (I have measured also over 100 ms once a minute or so, but this depends on uSD quality).

If you collect 512 bytes then you will have this uSD write delay every 25 seconds.

(BTW, I assume you know that all access to uSD are in chunks of 512 Bytes. If you write only 1 Byte, the file system has to read the old 512 Block, add the new Byte and write the 512 Byte block back to uSD)

If your real time requirements can tollerate the expected and possible delays, you have not to worry. If there are mission critical constraints, than you have to elevate these real-world interactions to high priority interrupt levels.

I don't think they can. And me and interrupts are not real friendly. LOL.
 
I just need 10 to 20 seconds as of right now.

In that case just buffer the data for the duration - 512 bytes /second - the T_3.5 easily has 10 KB for 20 seconds and depending on use has 200+KB free.


Then perform the writes when complete - and work on the write during logging later.
 
In that case just buffer the data for the duration - 512 bytes /second - the T_3.5 easily has 10 KB for 20 seconds and depending on use has 200+KB free.


Then perform the writes when complete - and work on the write during logging later.

That's what would love to do but am stuggling with that. lol Very new to programming. I can get the math and the if statements but everything else is rough. Plus I need the data in csv. format. Adding commas to a buffer along with new lines is killing me. LOL
 
Start with a plan and then make a first simple step.

Perhaps a structure to read all the raw data values into.

Then an array of the structures for subsequent readings.

Then just loop through the array of data in the structs and output as CSV.
 
I understand what you are saying 100 percent. Just don't know how to implement. Sorry, just not knowledgeable enough. Haven't worked with data gathering much at all.
 
I did a quick search for an example putting together a structure leading to an array of structures and found this that looks straight forward as a teachable example.

C-Programming-Tutorial/21-C-Structure.aspx


It ends with an array of employees. Ignoring the name array of characters - put in the 10 -20 values you gather for each sample. You can name them individually if they are that unique.
 
opps … completed with Edit above … wasn't done and hit enter and browser used that to "POST".

See what you can make of that - if you get it started expressing what you want myself or others here will help if needed to get the data recorded.

First step would be to sample and record the data to that array of STRUCT's then send all the data to Serial print formatted as CSV so you can see it when done.

Then the next step would be to open the SD file and send it there.
 
Ok the structure is already created.

Code:
struct statuses {
  volatile bool hasSync;
  uint16_t RPM;
  long longRPM;
  int mapADC;
  int baroADC;
  long MAP; //Has to be a long for PID calcs (Boost control)
  byte baro; //Barometric pressure is simply the inital MAP reading, taken before the engine is running. Alternatively, can be taken from an external sensor
  byte TPS; //The current TPS reading (0% - 100%)
  byte TPSlast; //The previous TPS reading
  unsigned long TPS_time; //The time the TPS sample was taken
  unsigned long TPSlast_time; //The time the previous TPS sample was taken
  byte tpsADC; //0-255 byte representation of the TPS
  byte tpsDOT;
  volatile int rpmDOT;
  byte VE;
  byte O2;
  byte O2_2;
  byte O2_3;
  byte O2_4;
  int coolant;
  int cltADC;
  int IAT;
  int iatADC;
  int batADC;
  int O2ADC;
  int O2_2ADC;
  int O2_3ADC;
  int O2_4ADC;
  int dwell;
  byte dwellCorrection; //The amount of correction being applied to the dwell time.
  byte battery10; //The current BRV in volts (multiplied by 10. Eg 12.5V = 125)
  int8_t advance; //Signed 8 bit as advance can now go negative (ATDC)
  byte corrections;
  int16_t TAEamount; //The amount of accleration enrichment currently being applied
  byte egoCorrection; //The amount of closed loop AFR enrichment currently being applied
  byte wueCorrection; //The amount of warmup enrichment currently being applied
  byte batCorrection; //The amount of battery voltage enrichment currently being applied
  byte iatCorrection; //The amount of inlet air temperature adjustment currently being applied
  byte launchCorrection; //The amount of correction being applied if launch control is active
  byte no2FuelCorrection;// The amount of correction being applied if nitrous is active
  byte no2TimingCorrection;//The amount of correction being applied if nitrous is active
  byte flexCorrection; //Amount of correction being applied to compensate for ethanol content
  byte flexIgnCorrection; //Amount of additional advance being applied based on flex
  byte afrTarget;
  byte idleDuty;
  bool fanOn; //Whether or not the fan is turned on
  volatile byte ethanolPct; //Ethanol reading (if enabled). 0 = No ethanol, 100 = pure ethanol. Eg E85 = 85.
  unsigned long TAEEndTime; //The target end time used whenever TAE is turned on
  volatile byte status1;
  volatile byte Status2;
  volatile byte spark;
  volatile byte spark2;
  byte engine;
  unsigned int PW1; //In uS
  unsigned int PW2; //In uS
  unsigned int PW3; //In uS
  unsigned int PW4; //In uS
  volatile byte runSecs; //Counter of seconds since cranking commenced (overflows at 255 obviously)
  volatile byte secl; //Continous
  volatile unsigned int loopsPerSecond;
  boolean launchingSoft; //True when in launch control soft limit mode
  boolean launchingHard; //True when in launch control hard limit mode
  uint16_t freeRAM;
  unsigned int clutchEngagedRPM;
  bool flatShiftingHard;
  volatile uint16_t startRevolutions; //A counter for how many revolutions have been completed since sync was achieved.
  uint16_t boostTarget;
  byte testOutputs;
  bool testActive;
  uint16_t boostDuty; //Percentage value * 100 to give 2 points of precision
  byte no2Duty1 = 0;
  byte no2Duty2 = 0;
  uint16_t nosTimer = 0;
  uint16_t nosTimer2 = 0;  
  byte idleLoad; //Either the current steps or current duty cycle for the idle control.
  uint16_t canin[16];   //16bit raw value of selected canin data for channel 0-15
  uint8_t current_caninchannel = 0; //start off at channel 0
  uint16_t crankRPM = 400; //The actual cranking RPM limit. Saves us multiplying it everytime from the config page
  int oilPress;
  int fuelPress;
  uint16_t nitrousPress;
  uint16_t solenoidPress;
  int crankCasePress;
  

  //Helpful bitwise operations:
  //Useful reference: http://playground.arduino.cc/Code/BitMath
  // y = (x >> n) & 1;    // n=0..15.  stores nth bit of x in y.  y becomes 0 or 1.
  // x &= ~(1 << n);      // forces nth bit of x to be 0.  all other bits left alone.
  // x |= (1 << n);       // forces nth bit of x to be 1.  all other bits left alone.

};

The statuses are accessed by "currentStatus.RPM" for example.
 
Status
Not open for further replies.
Back
Top