Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 17 of 17

Thread: Is using a "String" still a bad idea?

  1. #1
    Senior Member
    Join Date
    Feb 2015
    Posts
    146

    Is using a "String" still a bad idea?

    I have already been told, and read that using a "String" instead of a "string" (char array) in a microcontroller is a bad idea due the indeterminant amount of memory that the "String" will occupy.

    Is this still the case or has some magic occur in the compiler to make "String" a safe alternative to use in microcontroller code?

    The reason why I ask the question is that because I just started working with bluetooth with the aim of using for a Android based GUI/HMI for the CAN logger I have for the Teensy4.1.

    I was looking at the Sparkfun HM-1xx library and guess what, "String" declarations. Puzzled I asked Sparkfun if this was "OK".

    So then I was looking at the various Android apps and thought Virtuino looked interesting. I opened the VirtuinoCM library and guess what, "String" declarations.

    There is a positng in the forum about using Virtuino and creating your own library for instead of VirtuinoCM and the poster has "String" declarations.

    Did I miss the memo in which using "String" in microcontrollers at will is now a safe and will not cause any potential instability/intermittent bugs?

    If so when/how did this occur?

    Thanks

    Bruce

  2. #2
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    15,075
    Strings (core code) seem to have proper support to function well and good in general on Teensy ... as far as known and tested ... YMMV.

    sz_C strings are not risk free - but don't dynamically play games w/memory.

    As long as memory fits and functions

  3. #3
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,045
    Opinions vary regarding use of malloc / free and dangers of possible memory fragmentation.

    Some people always draw a very hard line and insist that no dynamic memory allocation is ever used in any way. If you're developing critical products like medical devices, aircraft controls, military weapons systems, you probably should be extremely cautious. Lives are on the line for those applications!

    For less critical applications, many people feel a more pragmatic approach of observing the actual memory usage under the heaviest anticipated use cases and allowing a safety margin of extra memory is good enough. While you can't prove in a mathematically certain way the worst case, you can estimate the odds. For many cases where you use much less than the total memory, it's easy to show the odds are very low, in many cases pretty much virtually zero. Usually for those sorts of applications, you accept a tiny chance of failure (eg, also no redundant power supplies, battery backups, other ways things can go wrong) and generally try to design the system to be able to recover as gracefully as possible if a problem does occur. Sometimes just showing the user clear feedback about a problem is most important.

    When using String, you can mitigate much of the risk by using the reserve(size); function on "long life" String variables. Especially for a String which stay around and gradually grows in size as you collect more data, reserving the maximum size up front greatly reduces the chances of memory fragmentation problems.

    Which way you choose really depends on a judgement call where you balance the consequence of a failure, an estimate of the risk, and the extra cost incurred. It's an easy answer for people on the internet to say everyone else should never take any risk, no matter how unlikely, even when the consequence isn't severe, and even if the cost of guarding against it is high. That is always the safe answer. It's an answer that feel virtuous. Whether that sort of answer really makes sense in the practical reality of your application is call you have to make.


    Quote Originally Posted by bvernham View Post
    Puzzled I asked Sparkfun if this was "OK".
    I'm curious to hear what Sparkfun says (if they reply). Hope you'll share.

  4. #4
    Senior Member
    Join Date
    Feb 2015
    Posts
    146
    Thanks for the reply's. I have asked both Sparkfun and Virtuino and no responses. Frankly this forum is the first place I have seen a response of any type so kudos.

    I am just trying to wrap my head around why use "String" instead of "string". Yes I know some of the handling of "String" is easier like for example "+" instead of "concat" but for example in this code snipit:
    Code:
     String get() {
          String returnLine = "";
    
          byte inByte;
          
          while ( Bluetooth.available() ) {
           
             inByte = Bluetooth.read(); 
             switch (inByte) {
    
                case '\n':   // end of text
                   input_line [input_pos] = 0;  // terminating null byte
    
                   // terminator reached! now process input_line
                   // reset buffer for next time
                   input_pos = 0; 
                   returnLine = input_line;
                   return returnLine; 
                   break;
                case '\r':   // discard carriage return
                   break;
                case '!':    // start of virtuino command
                   input_pos = 0;
                   input_line [input_pos++] = inByte;
                   break;
                case '$':    // end of virtuino command 
                   input_line [input_pos++] = inByte;
                   input_line [input_pos++] = 0;  // terminating null byte
    Yes, I know this is is not complete but just illustrative. All the handling of the incoming data is done with a "string" and even formatted with the '\0' terminator but then returned as a "String" instead of a "string"? Why?

    I am not understanding the any real or perceived advantages of "String" except it makes things "easier". But I do not understand how having two forms of the same values (the text) represented in two different ways "string" and "String" is easier.

    Just and observations from someone with only moderate coding skills.

    Thanks

    Bruce

  5. #5
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    1,070
    +1 on what Paul said. For some cases, the MISRA-C policy of "Dynamic heap memory allocation shall not be used" is best.

    > the odds are very low,

    But note that if there is any chance of someone trying to deliberately cause a problem, the odds go way up.

  6. #6
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,045
    Quote Originally Posted by jonr View Post
    if there is any chance of someone trying to deliberately cause a problem, the odds go way up.
    The same is often said of buffer overflow bugs with char array strings.

  7. #7
    For platforms other than Teensy, I sometimes use an intermediate policy of using malloc() to do one-time allocations of variables that exist "forever", such as communication buffers, but never use free(). There are two reasons I might do this instead of using static variable allocations. One is that I want the variable to be in the RAM assigned to heap/stack because it is faster (or slower) than the RAM used for static variables. The other reason is that buffer sizes can be arguments to configuration functions.

  8. #8
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    1,070
    What is the proper way to detect an out-of-memory failure of "new" or some std::string related statement (eg, S += "x") in a teensy program?

  9. #9
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    1,070
    I did some tests:

    Looks like "new" with no memory will return NULL which is non-conforming C++.

    String operations that can't allocate more memory generally quietly fail, with who knows what effect on your program. In some cases, it may crash/reset.

    The standard solution is exceptions, but it's not clear that these work on teensy.

    I didn't see errno change.

    With what I know (not much), if there are concerns about lack of memory and wrong program behavior, avoid Strings on a teensy.

  10. #10
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,045
    Quote Originally Posted by jonr View Post
    Looks like "new" with no memory will return NULL which is non-conforming C++.

    The standard solution is exceptions, but it's not clear that these work on teensy.
    C++ exceptions and RTTI are not supported on Teensy. All C++ code is compiled with flags "-fno-exceptions -fpermissive -fno-rtti".


    String operations that can't allocate more memory generally quietly fail, with who knows what effect on your program.
    String it supposed to drop data when it can't allocate memory, so you get a proper String variable which is blank.


    In some cases, it may crash/reset.
    Can you demonstrate any of those cases?

  11. #11
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    1,070
    > String it supposed to drop data when it can't allocate memory, so you get a proper String variable which is blank.

    Here is code that eventually drops data but doesn't result in a String variable that is blank. Then much later it crashes (yes, checking S2 for NULL would probably solve this). Teensy 4.0, 1.8.15, debug optimization.

    Code:
    void setup() {
      // put your setup code here, to run once:
    
      delay(2000);
    
      Serial.println("hello");
      
    }
    
    String S = "abc";
    
      
    void loop() {
     
      S += S;
    
      String *S2 = new String;
    
      *S2 = "x2--------------------------------------------------------------------------------------------------------------------";
      
      Serial.printf("%d\n", S.length());
      Serial.println(*S2);
    
      delay(5);
    
    }
    Last edited by jonr; 09-17-2021 at 06:24 PM.

  12. #12
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    1,623
    I'm confused ?!? What do you expect that this code should do? You are allocating memory in loop without ever releasing it and not checking if the allocation was successful. Since the Teensy doesn't have an infinite amount of memory this has to crash finally. And, of course this has nothing to do with String. It wouldcrash for every other type as well.

    Here some code with proper checking which doesn't crash
    Code:
    #include "Arduino.h"
    #include <cstdlib>
    #include <new>
    
    void setup()
    {
        while (!Serial){}
    
        if(CrashReport)
            Serial.println(CrashReport);
        // put your setup code here, to run once:
    
        Serial.println("hello");
    }
    
    String S = "abc";
    
    void loop()
    {
       S += S;
    
       String* S2 = new (std::nothrow) String();
       if (S2)
       {
            *S2 = "x2--------------------------------------------------------------------------------------------------------------------";
            Serial.println(*S2);
        }
        else
        {
           Serial.println("Out of memory");
           while (1) {}
        }
        Serial.printf("%d\n", S.length());
    
        Serial.println(millis());
    
        delay(10);
    }
    After some 20s it runs out of memory, detects this and gracefully stops...

    Code:
    x2--------------------------------------------------------------------------------------------------------------------
    196608
    21563
    x2--------------------------------------------------------------------------------------------------------------------
    196608
    21573
    x2--------------------------------------------------------------------------------------------------------------------
    196608
    21583
    Out of memory
    new usually throws if it can't alllocate memory which won't work here. You have to use new (std::nothrow) if you wan't it to return a nullptr in stead of trying to throw a bad alloc exception.

  13. #13
    Quote Originally Posted by luni View Post
    new usually throws if it can't alllocate memory which won't work here. You have to use new (std::nothrow) if you want it to return a nullptr in stead of trying to throw a bad alloc exception.
    @luni, good to know! I tried a similar test, but without std::nothrow, and found that S2 was never NULL.

  14. #14
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    1,623
    Yes this can be confusing. Maybe it is possible to change the default behavior somehow? Need to goolgle this on a rainy Sunday afternoon...

  15. #15
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    1,070
    > you can mitigate much of the risk by using the reserve(size)

    This leads to how does one safely use reserve()?

    > reserve(): A bad_alloc exception is thrown if the function needs to allocate storage and fails.

    But "C++ exceptions and RTTI are not supported on Teensy".

    I suggest that it's still true: reliability will typically decrease as you proceed along "static->stack->malloc()->String" for string memory allocation on a teensy.

  16. #16
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    1,623
    This leads to how does one safely use reserve()?
    Reserve returns false if it can't reserve. So just check the return value...

    reserve(): A bad_alloc exception is thrown if the function needs to allocate storage and fails.
    No, reserve uses malloc which can not throw. https://github.com/PaulStoffregen/co...tring.cpp#L130

    I would be really interested in an example where the correct usage of Strings crashes a Teensy. I was trying hard but couldn't crash it so far. Here some tests using reserve:

    Code:
    #include "Arduino.h"
    
    void panic(String err)
    {
        Serial.println(err);
        //while (true) yield();
    }
    
    void testStack()
    {
        // try to geneate 3 200kB strings on the stack.
        // storage will still be on the heap
        // The third trial should run out of memory
        Serial.println("Test Stack");
        String s1, s2, s3;
        if (!s1.reserve(200 * 1024)) panic(" Out of Memory s1");
        if (!s2.reserve(200 * 1024)) panic(" Out of Memory s2");
        if (!s3.reserve(200 * 1024)) panic(" Out of Memory s3");
    
        // strings go out of scope here and release the allocated memory
    }
    
    void testHeap()
    {
        // try to geneate 3 200kB strings on the heap.
        // storage will also be on the heap
        // The third trial should run out of memory
    
        Serial.println("Test Heap");
        String *s1 = new String();
        String *s2 = new String();
        String *s3 = new String();
    
        if (!s1->reserve(200 * 1024)) panic(" Out of Memory *s1");
        if (!s2->reserve(200 * 1024)) panic(" Out of Memory *s2");
        if (!s3->reserve(200 * 1024)) panic(" Out of Memory *s3");
    
        delete (s1);
        delete (s2);
        delete (s3);
    }
    
    void testRandom()
    {
        Serial.println("Test Fragmentation");
    
        constexpr int arrSize = 10;
        constexpr int min     = 1 * 1024;
        constexpr int max     = 100 * 1024;
    
        for (int loop = 0; loop < 100'000; loop++)
        {
            String strings[arrSize];
            for (int i = 0; i < arrSize; i++)
            {
                size_t strSize = random(min, max);
                String &s      = strings[i];
    
                s = "loop: " + String(loop) + " " + String(i) + " requested: " + String(strSize);
                if (s.reserve(strSize))
                {
                    s += " OK";
                } else
                {
                     s += " not enough memory";
                }
                Serial.println(strings[i]);
                delayMicroseconds(50); 
            }
        }
    
         Serial.println("Test Fragmentation done");
    }
    
    void setup()
    {
        while (!Serial) {}
    
        if (CrashReport)
            ;
    
        testStack();
        testHeap();
    
        testStack();
        testHeap();
    
        testRandom();
    }
    
    void loop()
    {
    }
    Prints:
    Code:
    Test Stack
     Out of Memory s3
    Test Heap
     Out of Memory *s3
    Test Stack
     Out of Memory s3
    Test Heap
     Out of Memory *s3
    Test Fragmentation
    loop: 0 0 requested: 64066 OK
    loop: 0 1 requested: 34555 OK
    loop: 0 2 requested: 17823 OK
    loop: 0 3 requested: 75488 OK
    loop: 0 4 requested: 59225 OK
    loop: 0 5 requested: 77953 OK
    loop: 0 6 requested: 28250 OK
    loop: 0 7 requested: 43649 OK
    loop: 0 8 requested: 36198 OK
    loop: 0 9 requested: 33185 OK
    loop: 1 0 requested: 46862 OK
    loop: 1 1 requested: 27996 OK
    loop: 1 2 requested: 36174 OK
    loop: 1 3 requested: 15028 OK
    loop: 1 4 requested: 25306 OK
    loop: 1 5 requested: 21740 OK
    loop: 1 6 requested: 60975 OK
    loop: 1 7 requested: 22517 OK
    loop: 1 8 requested: 95584 OK
    loop: 1 9 requested: 91006 OK
    loop: 2 0 requested: 34954 OK
    loop: 2 1 requested: 96749 OK
    loop: 2 2 requested: 93755 OK
    loop: 2 3 requested: 101066 OK
    loop: 2 4 requested: 60715 OK
    loop: 2 5 requested: 34517 OK
    loop: 2 6 requested: 63874 OK
    loop: 2 7 requested: 3689 OK
    loop: 2 8 requested: 32029 not enough memory
    loop: 2 9 requested: 94147 not enough memory
    loop: 3 0 requested: 19076 OK
    loop: 3 1 requested: 88083 OK
    loop: 3 2 requested: 59918 OK
    loop: 3 3 requested: 81062 OK
    ....

  17. #17
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    1,070
    Thanks for the info.

    So one shouldn't expect teensy Strings to act like a C++ std::string or according to Arduino documentation (where we find "reserve() Returns Nothing").

    @bvernham: this adds additional risks to using String in teensy code.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •