Char array behavior w/T3.2

Frankthetech

Frankthetech

Well-known member
Hi, so I'm just trying to get a handle on the array (char type) operation.
running this code

Code: 
char oneBuf[10]={"is one"};
char twoBuf[20]={"second34"};
char lstBuf[40]={"big array"};

void showdata(){
  Serial.print("address of oneBuf --> "); Serial.print(int(&oneBuf),HEX);
  Serial.print(" oneBuf = "); Serial.println(oneBuf);
  Serial.println();
 
  Serial.print("address of twoBuf --> "); Serial.print(int(&twoBuf),HEX);
  Serial.print(" twoBuf = "); Serial.println(twoBuf);
  Serial.println();
 
  Serial.print("address of lstBuf --> "); Serial.print(int(&lstBuf),HEX);
  Serial.print(" lstBuf = "); Serial.println(lstBuf);
}


void setup() {
  Serial.begin(115200);
  while(!Serial && (millis()<1000));
  Serial.println();
  showdata();
  strcat(oneBuf, twoBuf);   //put more chars into oneBuf then it can hold
  Serial.println();
  Serial.println("after buffer overflow");
  showdata();
}

void loop() {
  delay(10);
}

2 questions,
why does the address of the lstBuf[40] end up before the oneBuf[10]? oneBuf and twoBuf seem to be in order.
OK so maybe the compiler decides the order, I think.
and why with strcat(oneBuf, twoBuf); does twoBuf[20] end up with 34, and missing the -second- word?
how does twoBuf lose part of itself with a strcat copy?
 
Last edited:
First issue I see:
Code: 
strcat(oneBuf, twoBuf);   //put more chars into oneBuf then it can hold
  Serial.println();
  Serial.println("after buffer overflow");

strcat knows nothing about the size of oneBuf, so it will simply blindly copy over parts of memory outside of it's range.
 
If you want to guarantee the arrays are stored sequentially (one after another) in memory, they need to be inside another object like a struct or class. The compiler is free to place global objects in any order it likes.
With regards to the strcat issue the source and the destination are the same which is an undefined operation.
 
I was trying to copy more char's into oneBuf[10] from twoBuf[20] than it could hold, to see the effect it would have on oneBuf.
as expected strcat did what was asked of it and blindly added it to oneBuf (there by writing beyond the end of oneBuf[10])

So before I did the strcat this,

address of oneBuf --> 1FFF8744 oneBuf = is one

address of twoBuf --> 1FFF8750 twoBuf = second34


after strcat buffer overflow,

address of oneBuf --> 1FFF8744 oneBuf = is onesecond34

address of twoBuf --> 1FFF8750 twoBuf = 34


As you can see twoBuf lost the word (second) and retained (34)
oneBuf should have overwritten into twoBuf,
light comes on!
I see that it did as expected.
Sometimes you just need to talk about it.
Thanks
 
twoBuf does not immediately follow oneBuf in memory. 1FFF8750-1FFF8744 = 0xC = decimal 12 (not 10) which means there is a two-byte gap between the end of oneBuf and the beginning of twoBuf. Those two bytes could be, for example, a 16-bit integer which the strcat has overwritten with the letters 'n' and 'd'. If this was a part of a much larger program, you could have very mysterious and extremely hard-to-debug crashes.
You shouldn't treat two distinct memory objects as one.

Pete
 
Pete I agree, and I would never let this code be used. This was just an test of what could happen when you don't make sure there's room for the copy. The code has no two byte variable declared, just the buffers, could the 2 byte gap just be padding?
I use the following to check,

Code: 
if(strlen(oneBuf)+strlen(twoBuf)<sizeof(oneBuf)){
    Serial.println("buffer twoBuf will fit in buffer oneBuf");
    strcat(oneBuf, twoBuf);        //add it to
  }else{
    Serial.println("Warning!! buffer twoBuf will not fit in buffer oneBuf, no copy done");

How can I see where all the variables are in memory? short of printing all the address's to Serial.
Frank
 
Last edited:
The code has no two byte variable declared, could the 2 byte gap just be padding?
Click to expand...

Yes. You are using a 32 bit processor. The compiler will generally align all variables to a 32 bit bounaries unless you explicitly force it not to. Anything from 9 to 12 bytes of memory will generally take up 12 bytes. This will include items within an struct, the compiler can add padding withn that structure if it wishes. The one exception is if you have several small variables in a row.

uint8_t a;
uint32_t b;
uint8_t c;
uint32_t d;

Would probably take up 32 bytes of memory while

uint32_t a;
uint8_t b;
uint8_t c;
uint32_t d;

may only take 24 bytes because the two uint8_ts are packed together.

But this is all at the whim of the compiler. Depending on how it is optimising things they may both take 24 or both take 32.

Unless you have explicitly placed variables within a struct and then set a compiler directive to pack things within that struct in a specific way you should never assume anything about where the compiler is putting things. Especially when optimisation is turned on.

How can I see where all the variables are in memory?
Click to expand...

The linker generates a .sym file that gives you the memory addresses for all functions and variables.
It is in the build directory which can be hard to find at times. The easiest way I have found is to verify the project so that it brings up the teensy loader program. Select file->open and copy the directory name that it's pointing to.
Paste the directory name into the address of a file browser window.
You should find a <projectname>.ino.sym file there.

That said this isn't something you should normally need to look at unless you are trying to track down a particuarly nasty memory corruption bug.
 
Frankthetech said:
Pete I agree, and I would never let this code be used. This was just an test of what could happen
Click to expand...
Just about anything could happen in cases like this. Not sure your test will gain you any insight as there are many different
ways this could happen.

Example if you are using local variables for it, you would be trashing the stack. Maybe this wipes out the return address, or maybe local
variable that has the counts or indexes or ???

Frankthetech said:
How can I see where all the variables are in memory? short of printing all the address's to Serial.
Click to expand...
The addresses generated are also probably different depending on things like compiler settings and the like.

The best way to find the build directory is to go into the compiler settings and make sure that the verbose build options
are enabled:
1767625901253.png

The Show verbose output during... will show you the command lines that are generated which includes the paths
to where the build directory is.
 
Thanks, I got a better understanding on this now. Always learning new stuff.
I used this code to show how coping beyond the buffer size can cause issues.

Code: 
char oneBuf[12]={"firstbuf "};
char twoBuf[20]={"secondbuffer"};

void showLines(){
  if(strlen(oneBuf)+strlen(twoBuf)<sizeof(oneBuf)){
    Serial.println("buffer b will fit in buffer a");
  }else{
    Serial.println("Warning!! buffer b will not fit in buffer a");
  }
 
  char *adr = &oneBuf[0];
  Serial.print("oneBuf[12] address is --> ");
  Serial.println(int(adr), HEX);
  for(int x=0; x<12; x++){
    //Serial.print(x);
    Serial.print("address --> ");
    Serial.print(int(adr)+x, HEX);
    Serial.print(" contains -> ");
    Serial.print(adr[x], HEX);
    Serial.print(" = ");
    Serial.println(adr[x]);
  }
  Serial.println();
  adr = &twoBuf[0];
  Serial.print("twoBuf[20] address is --> ");
  Serial.println(int(adr), HEX);
  for(int x=0; x<20; x++){
    //Serial.print(x);
    Serial.print("address --> ");
    Serial.print(int(adr)+x, HEX);
    Serial.print(" contains -> ");
    Serial.print(adr[x], HEX);
    Serial.print(" = ");
    Serial.println(adr[x]);   
  }
}

void setup() {
  Serial.begin(115200);
  while(!Serial && (millis()<1000));
  Serial.println();
  Serial.println("-before using strcat on buffers-");
  showLines();
  strcat(oneBuf, twoBuf);   //put more chars into oneBuf then it can hold
  Serial.println();
  Serial.println("-----let's do it anyway... fubar!-----");
  Serial.println("-after strcat causing buffer overflow-");
  showLines();
  Serial.println("Done");
 
}

void loop() {
  delay(10);
}
Frank
 
Internally compilers will use hashtables for lots of things like names, constants and identifiers, so any arbitrary ordering could be down to the hash function on the day. Also optimization passes might order things by expected usage too...
 
Short of using a library (ie: Safestring) does anyone have ideas/examples about how to safely copy/cat buffer to buffer?
 
jmarsh said:
Use strncat to ensure the result doesn't overflow.
Click to expand...
Note with strncat, it may not work the way you expect it to...
Code: 
strncat
char * strncat ( char * destination, const char * source, size_t num );
Append characters from string
Appends the first num characters of source to destination, plus a terminating null-character.

If the length of the C string in source is less than num, only the content up to the terminating null-character is copied.
We probably want some more like strlcat...
Although I don't remember if this exists in the Arduino/Teensy stuff or not:

CoPilot generated this when I was asking about strlcat:
Code: 
#include <string.h>


size_t my_strlcat(char *dst, const char *src, size_t size) {
    size_t dlen = strnlen(dst, size);
    size_t slen = strlen(src);


    if (dlen == size) return size + slen; // No space to append


    size_t copy_len = size - dlen - 1;
    if (copy_len > 0) {
        strncat(dst, src, copy_len);
    }


    return dlen + slen;
}
 
Done with,
Code: 
strncat(oneBuf, twoBuf, (sizeof(oneBuf)-(strlen(oneBuf)+1)));
However, I can see an issue if either array is missing the nul terminator.
Will have to confirm the arrays are properly null termed before any strcpy/strcat operation.
Lots to consider.
 
Last edited:
jmarsh said:
Which is why the size should always take space for the nul terminator into account by subtracting 1.
Click to expand...
Yep! However even more, I think a lot of us expect that you should be able to do something like:
Frankthetech said:
Code: 
strncat(oneBuf, twoBuf, (sizeof(oneBuf)-(strlen(oneBuf)+1)));
Click to expand...
But the length is not the length of the destination field, as this would sort of expect, it instead is the max number
of characters to copy out of the Source buffer into the destination buffer. So if your Destination already has 10 characters within it
, the above code can override the buffer by 10 characters...

You need something more like: strncat(onebuf, twobuf, sizeof(onebuf) - strlen(onebuf) - 1);
 
This is not as straight forward as I first thought, I'm trying to better understand this kind of issue.
The reason I want know more is because in a rather large program I wrote (>5000 lines) running on T4.1 and is working 99.9% of the time.
I have the program log it's errors and once in a while the log shows this, maybe once or twice a year.

CrashReport:
A problem occurred at (system time) 11:12:1
Code was executing from address 0x8E24
CFSR: 82
(DACCVIOL) Data Access Violation
(MMARVALID) Accessed Address: 0x0 (nullptr)
Check code at 0x8E24 - very likely a bug!
Run "addr2line -e mysketch.ino.elf 0x8E24" for filename & line number.
Temperature inside the chip was 69.95 °C
Startup CPU clock speed is 396MHz
Reboot was caused by auto reboot after fault or bad interrupt detected

The system reboots and it's up again, but I have some time to look into this, so I am thinking it's a buffer overflow problem.
addr2line points to the arduinoJson library, Arduino\libraries\ArduinoJson\src/ArduinoJson/Memory/StringPool.hpp:89

I've been through the code many times and found nothing obvious. So now I will try to add some array buffer checks.

I'm thinking;
1: make sure that both arrays are null terminated
2: that A has enough room for the strncat B into A
3: check the result is also nulled

Find a way to add it to the code with out a large rewrite. That's doable, Right! Going to need more coffee.
 
Last edited:
To expand on jmasrh's comment - Have you checked through the code looking for any potential memory leaks?

Anything allocated using malloc or new should be released using free or delete.
If you call either of those and then either the variable goes out of scope or gets overwritten then the memory is lost forever. After a while there is no more memory to allocate, the next attempt to frequest more memory will return a null pointer. If you then try to use that pointer assuming it worked then you'd get the error you're seeing.
 
If you can provide the code snippet around where the exception happens, it would be easier for us to pinpoint the error. What data type is used as the input for the ArduinoJSON?, if it's a const string for some reason then the strings are copied to the JSONdoc and if that goes out of scope you get this problem otherwise it's a memory leak and preferably you could use a static JSON doc that allocates on the stack, or reuse a dynamic as a "memory pool"
 
Did you try running the command mentioned? addr2line -e mysketch.ino.elf 0x8E24

Most likely you need to edit this command unless your sketch is called mysketch.
And depending on host machine, you may need to search for addr2line command. On my Windows machine I have
copied one into a directory on my path so that part is easy.

Probably the version I have now in my path came from:
Code: 
C:\Users\kurte\AppData\Local\Arduino15\packages\teensy\tools\teensy-compile\11.3.1\arm\bin\arm-none-eabi-addr2line.exe
 
Hi Kurt,
I did try this, looks like it suggested the issue is in ArduinoJson library, likely my code calling it at some point.
It reports nullptr, not too sure what to look for. I'm not using malloc or new in my code but I expect it could be
called from a library.

CrashReport:
Code was executing from address 0x8E24
CFSR: 82
(DACCVIOL) Data Access Violation
(MMARVALID) Accessed Address: 0x0 (nullptr)
Check code at 0x8E24 - very likely a bug!

Running "addr2line -e C:\mypath\mysketch.ino.elf 0x8E24" for filename & line number.

addr2line reports the arduinoJson library, Arduino\libraries\ArduinoJson\src/ArduinoJson/Memory/StringPool.hpp:89
 
Last edited:
You must log in or register to reply here.
Back
Top