SdFat on GitHub

Status
Not open for further replies.
B

Bill Greiman

Well-known member
I just posted a new version of SdFat on GitHub https://github.com/greiman/SdFat/.

This version has a Teensy 3 SPI fix that Paul sent. It also has many other changes and several new examples.

I have only tested a few examples on Teensy 3. Various versions of the Teensy software fail in different ways with my use of USB Serial input. I don't have the patience to fight this problem.

Teensy 1.20RC2 seems to work somewhat.

Try the LowLatencyLogger example, it uses one of the new features. It is easily modifiable and does not use an RTOS or tricky ISR. I was able to log four analog pins on a Teensy 3.0 at 2500 Hz. I used a low cost class 4 SanDisk card.

type:
c - convert file to CSV
d - dump data to Serial
e - overrun error details
r - record data

Creating new file
Erasing all data
Logging - type any character to stop
Truncating file
File renamed: DATA08.BIN
Max block write usec: 203
Record time sec: 166.286
Sample count: 415716
Samples/sec: 2500.01
Overruns: 0
Click to expand...

Here is a bit of the output with floating pins. The first column is the time in microseconds.
time,adc0,adc1,adc2,adc3
6714002,247,180,139,78
6714401,125,109,89,68
6714802,147,140,180,110
6715201,109,219,46,74
6715602,154,78,11,130
6716002,50,96,128,154
6716401,245,224,116,119
6716801,171,166,116,100
6717201,145,120,116,106
6717601,253,136,124,87
Click to expand...

Here are the functions you need to modify for you needs.
Code: 
const uint8_t ADC_DIM = 4;
struct data_t {
  unsigned long time;
  unsigned short adc[ADC_DIM];
};

// Acquire a data record.
void acquireData(data_t* data) {
  data->time = micros();
  for (int i = 0; i < ADC_DIM; i++) {
    data->adc[i] = analogRead(i);
  }
}

// Print a data record.
void printData(Print* pr, data_t* data) {
  pr->print(data->time);
  for (int i = 0; i < ADC_DIM; i++) {
    pr->write(',');  
    pr->print(data->adc[i]);
  }
  pr->println();
}

// Print data header.
void printHeader(Print* pr) {
  pr->print(F("time"));
  for (int i = 0; i < ADC_DIM; i++) {
    pr->print(F(",adc"));
    pr->print(i);
  }
  pr->println();
}
 
Hi Bill. Have you seen my recent work on SPI transactions?

Any idea how this might work with SdFat? It seems you don't include SPI.h anywhere, which is great for making a totally stand-alone library. But how can SdFat coordinate access to the SPI bus with other code which uses the SPI library?
 
Last edited:
You can use the standard SPI library but it is really slow. Edit SdFatConfig.h
Code: 
/**
 * Force use of Arduino Standard SPI library if USE_ARDUINO_SPI_LIBRARY
 * is nonzero.
 */
#define USE_ARDUINO_SPI_LIBRARY 0

But how can SdFat coordinate access to the SPI bus with other code which uses the SPI library?
Click to expand...

In practice it's hopeless. When you read from an SD card there is no way to know when data will arrive. You just read until you get a start of data token. The same is true for write block, you send the write command and then send FF and see what comes back. when you get a ready token, you send a start data token followed by the block. Cards can go busy for up to 250 ms in single block write mode and that's what is often used in SdFat.

Years ago I thought something like your approach might work.

The real answer is chips like the STM32 with multiple SPI controllers. I don't use Teensy or Due for my projects since I want to configure multiple SPI and I2C controllers on the pins I choose. I really like SDIO on STM32, I now get up to 20 MB/sec read and write with my new FAT, not SdFat, library.

Edit: SDIO controllers are magic. They do all he stuff that the SPI library does plus most of what the SD card library does. In an RTOS, you set up a many block transfer and sleep until the data is transferred. The chip firmware is really impressive.
 
Last edited:
Bill Greiman said:
You can use the standard SPI library but it is really slow. Edit SdFatConfig.h
In practice it's hopeless. When you read from an SD card there is no way to know when data will arrive. You just read until you get a start of data token. The same is true for write block, you send the write command and then send FF and see what comes back. when you get a ready token, you send a start data token followed by the block. Cards can go busy for up to 250 ms in single block write mode and that's what is often used in SdFat.
Click to expand...

That's a very good point, and one I didn't really consider in my previous patching on Arduino's SD library (based on an old SdFat verson).

In that loop, waiting on the card, the SD library should really use SPI.endTransaction() when the card is busy, then SPI.beginTransaction() before checking again. I'll make this change soon....


You can use the standard SPI library but it is really slow. Edit SdFatConfig.h
Click to expand...

Would you be willing to consider to add another option to use only the SPI transaction functions, together with your fast native code for the actual SPI data movement?
 
Would you be willing to consider to add another option to use only the SPI transaction functions, together with your fast native code for the actual SPI data movement?
Click to expand...

When I saw your "SPI Transactions in Arduino", I decided "If You Can't Say Anything Nice, Don't Say Anything". Since you asked, I have no interest in SPI transaction functions.

Masking interrupts to share a hardware controller with non-interrupt code is a terrible idea. Mix that with a preemptive priority based scheduler and it becomes even worse.

What problem does this idea solve? If I need SPI in an ISR, I want reasonable interrupt latency. Allowing random libraries to control interrupt latency for my ISR is unacceptable. No sane RTOS has a feature like this.

I recently wrote a RAM disk file system that uses SRAM and FRAM chips. I use SPI writes and reads up to the size of an allocation unit which can be 32 KB. Will this be OK with your SPI transaction functions? It will send interrupt latency sky high for ISRs that use SPI.

Due and Teensy only provide access to one SPI controller, this is a poor choice for a Cortex M board. The Due chip has four SPIs and I think the Teensy chip has two or you should have picked a chip with two or more. SPI transaction functions won't fix bad hardware design.

I have been using a STM32 Nucleo board for a recent project http://www.st.com/web/catalog/tools/FM116/SC959/SS1532/LN1847/PF260320. This board has three SPIs and three I2Cs. It has headers for Arduino shields and STDIO. Too bad Due is not more like this. It only costs $10.33 since ST subsidizes it.

Note that the Morph headers are double ended so you can put a base board under the Nucleo. I made a board with an SD socket and battery backup for the RTC (I added HSE and RTC crystals in the provided area on Nucleo). Others have added USB and Enet connectors.
 
Last edited:
There is a standard way to handle this in the embedded world. It's sometimes called "deferred interrupt processing".

You only give a semaphore in the ISR. This wakes a thread that uses standard RTOS resource sharing mechanisms. It's flexible since you can handle the fast high priority part of the response in the ISR and defer the rest to the thread.

Here is a FreeRTOS example.
Code: 
// Somewhere in main() :
SemaphoreHandle_t event_signal;
vSemaphoreCreateBinary( event_signal ); // Create the semaphore
xSemaphoreTake(event_signal, 0);        // Take semaphore after creating it.
 
 
void System_Interrupt()
{
    xSemaphoreGiveFromISR(event_signal);
}
 
void system_interrupt_task()
{
    while(1) {
        if(xSemaphoreTake(event_signal, 9999999)) {
            // Process the interrupt
        }
    }
}

This idea dates back to the early 1970s.
 
In my upcoming transition to Arduino 1.5.x, which has support for different libraries to be used for different boards, I'm hoping to have all Teensy boards use the latest SdFat, rather than Arduino's SD library based on a very old version of you wrote years ago.

I suppose I could patch SdFat and maintain a separate copy, if you absolutely won't accept a non-RTOS hardware access arbitration method. Arduino already applied their own patches with an ancient version of your library, and I'm already doing the same with their code. It works, but it creates a huge obstacle to keeping up with the latest versions you publish.

Then again, if you're moving in an RTOS-only direction and only supporting the STM32 chips, maybe maintaining a separate code base is a better long-term solution? Or maybe I've just misunderstood your point? It seems SdFat still has a LOT of code for many different platforms, even bitbashing for chips without any SPI hardware. I guess I'm a bit confused about whether SdFat should be used on Teensy?
 
First off, I really appreciate the work that both of you do, with building up stuff for the rest of us to use. I do try to help out as well as I can...

Note: I have also been around the block a few times, starting to work as a Software Engineer back in the 70s... Worked on the internals of a few different OS's...

But I am a bit confused here and hoping I am not stepping onto anyone's toes. But I personally don't understand how the generic concept of deferred interrupt processing is going to help me in the simple cases of the Teensy or Arduino?

I am trying to be pragmatic here and adapt to whatever the agreed upon ways to solve things as long as it works OK...

Example of a project I am toying with: Well monitor system. Hardware may include:

Teensy 3.1 - Really like these little chips, probably mounted on one of my breakout boards. Plus all of the sensors...

Adafruit 2.8" TFT shield - Use SPI ILI9341 - using fast library ILI9341_t library which directly manipulates the SPI registers.

Touch screen on the above display - (Adafruit_stmpe) Currently uses SPI library to communicate with onboard chip. SPI library mucks with SPI registers.

Maybe nRF24L01 wireless connection, which also uses SPI - Not sure what library yet...

And SDCard on the TFT shield - Probably SDFat, which also directly manipulate the SPI registers...

So If not Transactions? How should programs setup such that all of these devices work on the one SPI buss? Note: in my case I will probably only use SDCard to do some periodic logging of data and probably not that frequently. So I may be able to get away with simply Opening SDFat when I need it, do everything I need with it and then close it down and do what is necessary to recover the SPI state...

But it would be nice if there was some clean easy way, that is consistent with normal Arduino stuff, which currently has no concepts of threads, tasks, semaphores... Also it would be great if the mechanism was light weight, ideally only doing stuff when necessary. i.e. if only one SPI device call SPI.begin(), then maybe almost nothing happens...

Again I appreciate all of the hard work you guys are doing!

Kurt
 
I can't believe you think masking interrupts is a suitable solution to sharing. A little of this type stuff was done with machines like the IBM Stretch in the early 1960s. The Stretch was the first machine with vectored interrupts and a mask register. Better solutions were soon found that were independent of the interrupt controller.

You don't need an RTOS to solve this problem but your solution is really stone age.

Here is an old solution. Have the ISR call a simple dispatcher with it's interrupt number. If the SPI is free, execute the service code. If the SPI is reserved set a pend bit and execute the service code when a SPI yield or SPI release function is called. This idea is more general than your solution and you don't need maskable interrupts.

SdFat has a limited future. I will soon release a new FAT library that will work with any block device. SdFat just evolved randomly starting in 2008 with a 168. The new library can be used with multiple types of block devices in the same program. The FAT part is device and OS independent. The sharing stuff would be in the block device code. It is designed to work with bare hardware, Arduino, or an RTOS. I have tested it with SD on SPI, SD on SDIO, and SPI SRAM/FRAM. I hope to support USB mass storage.

Looks like SdFat needs four calls, a reserve SPI in chipSelectLow, a release SPI in chipSelectHigh, a SPI yield call in wait busy, and in read block waiting for data start token. I guess I could I could put these calls in, maybe as weak or with an #if, #endif.
 
Last edited:
Bill Greiman said:
Here is an old solution. Have the ISR call a simple dispatcher with it's interrupt number. If the SPI is free, execute the service code. If the SPI is reserved set a pend bit and execute the service code when a SPI yield or SPI release function is called. This idea is more general than your solution and you don't need maskable interrupts.
Click to expand...

I don't see what advantage this has over simply masking the specific interrupt(s) that are known to do SPI access. Either way, you don't get to access the SPI bus until the other code is finished.

While this approach would avoid the need to register interrupts, so beginTransaction() knows which ones to mask, it requires two code paths (if the SPI is available vs busy at the time of interrupt) for anyone implementing a library for a SPI chip that requests service by an interrupt.

I should mention this transaction stuff doesn't necessarily have to be implemented with interrupt masking. Under FreeRTOS, a library for SPI could implement this API using a semaphore or mutex, if it also implement's Arduino's attachInterrupt() function as a request to signal the semaphore or event that causes a task to run. But the API is specifically designed to NOT require authors using SPI to implement 2 different code paths, depending on whether the SPI is available or busy.

Looks like SdFat needs four calls, a reserve SPI in chipSelectLow, a release SPI in chipSelectHigh, a SPI yield call in wait busy, and in read block waiting for data start token. I guess I could I could put these calls in, maybe as weak or with an #if, #endif.
Click to expand...

The intention is to use "#ifdef SPI_HAS_TRANSACTION", which will be defined by newer versions of the SPI library supporting this feature. So far, Teensyduino 1.20-rc2 is the only software supporting this feature. It's a milestone for Arduino 1.5.8. Maybe it'd be best to consider this when Cristian merges it into the 1.5.x branch, or when he releases Arduino 1.5.8.
 
You don't need two paths. There is one service routine. It is either executed immediately or later when the resource is free. This solution is far simpler than your solution. It is easy to generalize.

The ISR often has a macro that checks if the resource is free and then either calls the service routine immediately or calls a function to defer execution of the service routine. The same code is executed in either case.

Guess you never had an OS course, deferred interrupt processing is a standard topic usually covered early, around lecture 6.

Looks like you are a hard core not invented here guy.

Either way, you don't get to access the SPI bus until the other code is finished.
Click to expand...

Wrong! you can execute part of the ISR immediately and defer the part that needs a resource. This covered in your OS course also.
 
Last edited:
In any high school debate class - if you personally try to insult someone, you lose the argument! But I guess you never took that course either
Click to expand...

Your right. I have a big hot button about solving generic system architecture problems with hardware specific tricks. Over the years I had several hardware engineers on my projects who insisted on building hardware solutions for problems that long ago had been solve with generic software algorithms. I had some irrational moments with engineers who would not accept my decision to not build their hardware. I still go into my "not invented here hardware engineer" mode real fast.

I apologize, I didn't mean to insult Paul.

I should have explained why operating systems solve the problem of shared resources/critical sections in interrupt routines the way they do. Also the solution can be simple, lightweight, and doesn't need to include any of the baggage people associate with operating systems.
 
I understand your point, or at least I think I do. I know you have the best of intentions. I'm not offended, really. Often on technical matters, brutal honesty is the best way.

Unfortunately, it's too late to reopen this debate. This was all discussed in great detail back in April and May, on the Arduino Developers Mailing List. Deferred execution and conventional operating systems were discussed, as were numerous other topics, including an idea for event callbacks. Test code and benchmarks were written, which showed pretty conclusively that more general approaches with callbacks suffered terrible performance on 8 bit AVR. During this lengthy discussion, API was changed from my original ideas, mainly by a proposal by Matthijs Mooikman. His approach to SPI setting really was much better than mine.

Ultimately, Cristian Maglie (technical lead for Arduino and the man who ultimately decides what becomes the Arduino API) made a final decision about all this stuff on May 22.

The time to propose a lightweight operating system-like approach, and show that it can actually be lightweight, was months ago.
 
Then again, a lightweight operating system approach could (maybe) still be implemented, if it can be shoehorned into this transaction API, perhaps with some extra extensions.

Before these functions, there simply wasn't any way to "know" when code using the SPI functions needed to gain and release access to the SPI bus. SPI.beginTransaction() and SPI.endTransaction() add that information to the code. If someone felt strongly about implementing the Arduino API on top of an RTOS or full operating system, or something else with operating system like features, perhaps these functions could be used to signal semaphores instead of masking interrupts. Perhaps extra stuff would be needed inside the function given to attachInterrupt?

So far, the simple reality is nobody other than me as been willing to actually put real work into this SPI sharing problem (though Mikael Patel did offer a lot of insight from his experience developing his Cosa library, which implements something similar). After many weeks of public discussion, where several people did offer many suggestions and a couple people even backed those suggestions up with some test code & benchmarks, the net result was no actual progress for nearly 2 months when I didn't have time. There was plenty of opportunity for others to step up with an implementation, but nobody did.

Still, if someone were to come up with a superior approach, with a working implementation that truly is lightweight, and doesn't impose burdensome requitements to porting the hundreds of existing Arduino libraries, I'll seriously consider abandoning the code I've written and go with something that's better. I can't speak with certainty for the Arduino folks, but I believe they genuinely want to do what's best for their huge userbase.
 
Paul,

This code shows the principle used by deferred interrupt processing.

I just hard coded an example I could use with one ISR. The implementation would need a way to register a deferred ISR function in a table etc...

The deferred part of the ISR is executed when endTransaction() is called.

Code: 
// True when ISR function waiting for SPI
volatile bool spiDeferred = false;

// True when library is using SPI
volatile bool spiReserved = false;

void beginTransaction() {
  spiReserved = true;
}

void endTransaction() {
  cli();
  spiReserved = false;
  if (spiDeferred) {
    spiFcn();
    spiDeferred = false;
  }
  sei();
}

ISR(ANY_vect) {
  // Could do urgent stuff that doesn't access SPI here.
  
  // Hide the final form of this in a macro.
  if (spiReserved) {
    spiDeferred = true;
  } else {
    spiFcn();
  }
}

// ISR function that accesses SPI.
void spiFcn() {
  // Access SPI here.
}
 
Last edited:
Yes, I see the idea. This does have the advantage of letting the interrupt do other stuff immediately and defer the SPI work until the bus is later. But is that important?
 
Yes, I see the idea. This does have the advantage of letting the interrupt do other stuff immediately and defer the SPI work until the bus is later. But is that important?
Click to expand...

In the embedded world it is very important. The main reason for an ISR may be to access hardware that is not shared. The deferred part accesses hardware or memory that is shared with non-interrupt code and is less time critical. It is common for the deferred part to run at a different interrupt priority so other high priority interrupts can happen.

Deferred interrupt handling gets really bizarre in some systems, Linux has softirqs, tasklets, and work queues.

Too bad there isn't a vision for how Arduino needs to evolve. The SPI sharing problem is symptom of what users need, a preemptive scheduler is in the future. The solution to SPI sharing should be a step in the path to the future.

Old timers say an RTOS is way beyond what users can understand. I see young engineers at UCB doing things for class projects that I never thought would be possible, it only takes a few days for these students to abandon bare hardware.

Here is another maybe true statement from recent news.
An average six-year-old has the same knowledge of technology as a 45-year old
Click to expand...
 
Last edited:
Bill Greiman said:
In the embedded world it is very important. The main reason for an ISR may be to access hardware that is not shared. The deferred part accesses hardware or memory that is shared with non-interrupt code and is less time critical.
Click to expand...

Are there any specific SPI chips that really need this?

In other words, I think I understand the idea. Maybe? If the interrupt is serviced immediately, then you could do something else right away. I'm just having a hard time imagining how any SPI chip I've used would need that. For example, W5200, CC3000, nRF8001, RFM69 all involve immediately talking to the chip to find out why it asserted its interrupt signal.

I would really like to understand this better, and I believe looking at a real example would make this more tangible for me to understand. Are there any Arduino libraries that try to accomplish this sort of thing today? Such a library would really illuminate the need for this functionality in Arduino, especially if it does through quite a bit of dirty trickery to emulate the missing RTOS features, better than many thousands of abstract arguments ever could.

Non-Arduino code might also help make this clearer, but if this argument is ever going to made convincingly to the Arduino devs, it's going to take a pretty clear example that shows a clear need independent of the requirements of some particular software system like FreeRTOS or Linux.

It is common for the deferred part to run at a different interrupt priority so other high priority interrupts can happen.

Deferred interrupt handling gets really bizarre in some systems, Linux has softirqs, tasklets, and work queues.

Too bad there isn't a vision for how Arduino needs to evolve. The SPI sharing problem is symptom of what users need, a preemptive scheduler is in the future. The solution to SPI sharing should be a step in the path to the future.
Click to expand...

Cristian Maglie, Massimo Banzi and perhaps other people in Arduino are the ones who decide where Arduino goes as a platform. My influence is very small. Often they reject my ideas.

RTOS and operating system style scheduling were discussed back in April. They decided not to move to an RTOS model. If I had built this using an RTOS model, against their decision, Arduino would not accept my pull request. My main concern is to solve this problem now, with the growing number of conflicting libraries that real people are suffering with using today. To help the greatest number of people (not just PJRC customers), I really need to work within the scope of what they'll accept.

Long term, perhaps Arduino will indeed move to an RTOS model? That's up to Cristian & Massimo.

I might move Teensy in that direction over time, even if Arduino doesn't. I'm not looking to diverge much from Arduino's API, nor do I want to introduce sudden changes that cause compatibility problems.
 
Last edited:
Are there any specific SPI chips that really need this?
Click to expand...
It reason is more general. You don't know why an application uses an interrupt. Maybe the interrupt was caused by a pin change and knowing the time of the event as accurately as possible is important. The SPI part may access a sensor where time is not as important.

I could give lots of other examples but it's not worth trying to convince Arduino management. It's their business/technical decision.

Adding OS features like sharing in random bits and pieces is the wrong approach.

I read the discussion that lead to the SPI sharing decision. I could never convince these people that when users need synchronization, sharing, and threads, it's time to use an RTOS as your foundation. You just run setup() and loop() in the idle thread. Old user programs don't change.

Here is a real example using a tiny micro-kernel, Nil RTOS, written by Giovanni Di Sirio. This shows how lean a micro kernel can be.

The first program is the Arduino Blink example which uses 1084 bytes of flash.

The second program adds two threads, semaphores, and thread sleep. It takes 1,594 bytes of flash. Old Arduino programs would require no extra flash since no kernel code would be linked. If you don't use the RTOS features you just have good old setup() and loop().

In the future I will bite my tongue and try to live with whatever the Arduino company does.

Arduino Blink example. Binary sketch size: 1,084 bytes (of a 30,720 byte maximum)
Code: 
/*
  Blink
  Turns on an LED on for one second, then off for one second, repeatedly.
 
  This example code is in the public domain.
 */
 
// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
int led = 13;

// the setup routine runs once when you press reset:
void setup() {                
  // initialize the digital pin as an output.
  pinMode(led, OUTPUT);     
}

// the loop routine runs over and over again forever:
void loop() {
  digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(1000);               // wait for a second
  digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  delay(1000);               // wait for a second
}

RTOS Demo. Binary sketch size: 1,594 bytes (of a 30,720 byte maximum)
Code: 
/*
 * Example to demonstrate thread definition, semaphores, and thread sleep.
 */
#include <NilRTOS.h>
#include <DigitalIO.h>

// The LED is attached to pin 13 on Arduino.
DigitalPin<13> pin13(OUTPUT);

// Declare a semaphore with an inital counter value of zero.
SEMAPHORE_DECL(sem, 0);
//------------------------------------------------------------------------------
/*
 * Thread 1, turn the LED off when signalled by thread 2.
 */
// Declare a stack with 128 bytes beyond context switch and interrupt needs.
NIL_WORKING_AREA(waThread1, 128);

// Declare the thread function for thread 1.
NIL_THREAD(Thread1, arg) {
  while (TRUE) {
    
    // Wait for signal from thread 2.
    nilSemWait(&sem);
    
    // Turn LED off.
    pin13 = 0;
  }
}
//------------------------------------------------------------------------------
/*
 * Thread 2, turn the LED on and signal thread 1 to turn the LED off.
 */
// Declare a stack with 128 bytes beyond context switch and interrupt needs. 
NIL_WORKING_AREA(waThread2, 128);

// Declare the thread function for thread 2.
NIL_THREAD(Thread2, arg) {
  
  while (TRUE) {
    // Turn LED on.
    pin13 = 1;
    
    // Sleep for 200 milliseconds.
    nilThdSleepMilliseconds(200);
    
    // Signal thread 1 to turn LED off.
    nilSemSignal(&sem);
    
    // Sleep for 200 milliseconds.   
    nilThdSleepMilliseconds(200);
  }
}
//------------------------------------------------------------------------------
/*
 * Threads static table, one entry per thread.  A thread's priority is
 * determined by its position in the table with highest priority first.
 * 
 * These threads start with a null argument.  A thread's name may also
 * be null to save RAM since the name is currently not used.
 */
NIL_THREADS_TABLE_BEGIN()
NIL_THREADS_TABLE_ENTRY("thread1", Thread1, NULL, waThread1, sizeof(waThread1))
NIL_THREADS_TABLE_ENTRY("thread2", Thread2, NULL, waThread2, sizeof(waThread2))
NIL_THREADS_TABLE_END()
//------------------------------------------------------------------------------
void setup() {
  // Start Nil RTOS.
  nilSysBegin();
}
//------------------------------------------------------------------------------
// Loop is the idle thread.  The idle thread must not invoke any 
// kernel primitive able to change its state to not runnable.
void loop() {
  // Not used.
}
 
Over the last 6 years, I've learned a bit about what types of arguments do and don't work when talking with the Arduino devs.

No matter how logically sound, abstract proposals that have benefits like "more general way" never work. Arguing based on design patterns in other systems is also pretty much pointless with Arduino. I've seen many people make these types of proposals, always ending in failure and frustration. When you can't give specific examples of real applications and libraries that are struggling due to Arduino's limitations, or show groups of absolute novices confused by something, it's pretty much pointless to even try convincing them of anything.

For Teensy, I mostly follow Arduino's API. I do hear requests for RTOS support pretty regularly, so I certainly do have my eye on someday supporting an RTOS on Teensy. Obviously the Arduino devs have heard a lot of similar requests, which probably led them to develop the Scheduler library for Arduino Due. But there's a large gap between merely publishing a library and creating a well integrated system that delivers a satisfying experience.

You may have noticed Teensyduino has pervasive use of yield() in all functions that block. My recent patches to the Ethernet library have added yield(). I intend to do the same in SD before releasing 1.20, and other libraries over time. Someday I'm going to use this stuff to provide more concurrency. I'm still not sure if I want to pursue a RTOS approach or go with event callbacks, or perhaps a mix of both.

I could use constructive input on how to slowly lay more RTOS and concurrency groundwork. But you need to understand my intention at this point is only to slowly and gradually lay the groundwork, to make this more feasible in the future. In the meantime, I do intend to keep mostly following Arduino's API, and even through it seriously tries my patience and consumes too much time, I do intend to keep trying to collaborate with them. Nearly everyone I've met who loves RTOSs hates Arduino, which makes constructive conversation elusive.
 
No matter how logically sound, abstract proposals that have benefits like "more general way" never work.
Click to expand...
I understand and agree. I only put the RTOS example in to demonstrate to you how lean the technology can be. It's not a technology problem.

Nearly everyone I've met who loves RTOSs hates Arduino, which makes constructive conversation elusive.
Click to expand...
You can't argue this issue or even demonstrate with concrete examples. It's an experience that an individual must have. It's like the first time I flew with a friend in an aerobatic plane. I could never have guessed what it's like, the extreme G-forces, the world spinning around you.

It amuses me when people say I need OS features but don't want an OS, I will use a simple scheduler and yield. You don't need to choose. The yield thing is just round-robin with yield and most RTOSs support it. No one would say I don't want an OS on a smart phone. What makes the smartphone great is the richness of the OS.

You will never get a satisfactory result by evolving Arduino. You need a rich modern environment like the smart phone so users decide what bits they need.

I bet if these guys saw an automobile in 1900 they would have meetings on how to breed horses with car like features. No Steve Jobs need apply at Arduino.

The key to introducing modern techniques to hobby computing is not selecting an RTOS. Its the education effort that will be needed so the user experience is great. The key is a great teacher. You will need to provide a more user friendly API than used in my example. Some new tools will be needed in the IDE.

The UCB EE department had a mediocre embedded systems program for decades. Then Edward Lee took over http://www.eecs.berkeley.edu/Faculty/Homepages/lee.html. The change was overwhelming, you can't believe how excited students are.

Edit: Here is a nine university program he started, fun stuff, not dull ohms law style engineering education.
The TerraSwarm Research Center, launched on January 15, 2013, is addressing the huge potential (and associated risks) of pervasive integration of smart, networked sensors and actuators into our connected world. The center is funded by the STARnet phase of the Focus Center Research Program (FCRP) administered by the Semiconductor Research Corporation (SRC). Funding comes from the Defense Advanced Research Projects Agency (DARPA) and the SRC industry partners, including Applied Materials, GLOBALFOUNDRIES, IBM, Intel Corporation, Micron Technology, Raytheon, Texas Instruments, and United Technologies.
Click to expand...
 
Last edited:
Paul,

Here is another thing that bothered me about the shared SPI bus solution. RTOSs that I know about don't provide sharing protection in the SPI driver. The driver is the wrong place for this feature.

Resource sharing is much too complex for this type of approach to work. Some poor user will call yield while he has the SPI bus reserved and a deadlock will occur or another "loop" will also do a beginTransaction followed by an endTransaction and interrupts will have access to the SPI bus when they shouldn't.

This piecemeal approach won't work. The pieces won't fit together. All this stuff was solved more than 40 years ago.
 
Again don't mean to get in the middle of this. Personally I am not sure there is a right answer to many of these issues, as it depends on how you weight the tradeoffs. So I try to stay pragmatic. But I hope the two of you plus others can hopefully drag the Arduino people in a reasonable direction...

Kurt
 
Last edited:
http://www.dilbert.com/strips/comic/2014-08-10/
Again don't mean to get in the middle of this. Personally I am not sure there is a right answer to many of these issues, as it depends on how you weight the tradeoffs. So I try to stay pragmatic. But I hope the two of you plus others can hopefully drag the Arduino people in a reasonable direction...
Click to expand...

Users like you are the reason I get upset about this issue. You should be more involved so your needs are satisfied.

I don't think the Arduino company has the technical skill to deal with sophisticated software for embedded systems. There are many approaches for implementing any embedded system and I think Arduino should provide general tools and let you choose the best approach for your application, skill level, and programming tastes.

I sense a vacuum at Arduino. I have known and worked with some of the best people in this field. I don't see a leader with the right experience at Arduino.

I started developing operating systems soon after getting a PhD in particle physics. I got a job in in a physics group that received a grant to buy the first unclassified super computer, serial number two. It came without an OS and I was in a group of about 25 people formed to write an OS. It was a baptism in fire. The OS ran in 20 small 12-bit 4K word processors so the the main CPUs could run user code. I spent day and night tracking sharing problems.

Later I was involved with one of the first uses of an RTOS in physics. One of my colleagues, Dave Wilner, formed a company and developed the system used in all the Mars Rovers. The ARINC 653 certified version is use for avionics in big military and commercial aircraft.

If I were doing your project, I would start with the version of ChibiOS that I ported to Teensy 3.0. It many no longer run on current teensy releases and might not run on Teensy 3.1. I would be willing to fix the ChibiOS port if necessary and provide assistance if you want to try an RTOS solution.

Paul says a concrete example would be helpful. It would be a good exercise to try ChibiOS/RT for your problem.

Every time I look at the Arduino solution, I find problems. Paul's SPI fix and the Arduino scheduler library just won't work. Paul is putting yield and SPI transaction stuff in SdFat.

Here is why the Arduino solution won't work. Assume a user writes two processes and uses the Scheduler Library. The first process reads a file and the second process writes a file. If one of Paul's yield calls happens during an SdFat call in the first process, the second process will run and the SdFat's block cache may be changed. When execution returns to first process, the wrong cache block will be used and I will get a problem report for SdFat. You can't just put yield calls in a program that accesses global data.

Sharing something like SdFat requires at least two Mutexes, one for SPI and one for the SD global data structures. The Arduino solution just doesn't cut it.

Why Mutex and not Semaphore?
A mutex is essentially the same thing as a binary semaphore and sometimes uses the same basic implementation. The differences between them are:

Mutexes have a concept of an owner, which is the process that locked the mutex. Only the process that locked the mutex can unlock it. In contrast, a semaphore has no concept of an owner. Any process can unlock a semaphore.

Unlike semaphores, mutexes provide priority inversion safety. Since the mutex knows its current owner, it is possible to promote the priority of the owner whenever a higher-priority task starts waiting on the mutex.

Mutexes also provide deletion safety, where the process holding the mutex cannot be accidentally deleted. Semaphores do not provide this.
Click to expand...

Here is my view of how the "less than perfect" Arduino solution happens http://www.dilbert.com/strips/comic/2014-08-10/
 
Last edited:
Status
Not open for further replies.
Back
Top