Best way to handle large http POST data

Status
Not open for further replies.

turbo2ltr

Active member
Anyone have any guidance on how to handle large HTTP Posts using the Wiz850io?

I currently have code that will handle GET requests for any file off the SD card. One intent of this project is to allow editing of the configuration INI file on the SD card through the web interface, and save it back to the card (backing up the original)

I now need to post back the contents of the editor page. While I have no doubt I can do it, I would appreciate any high-level hints for the best method for handling the large amount of data. And when I say large, lets say (hopefully) under 20kb

I could create a fixed size buffer...but I'd need one for each socket which would largely go unused and suck up most of my memory.
I could manually malloc a buffer which seems the better way, based on the content-size header.

I know there are ramifications of each when it comes to memory allocation though I'm not smart enough to understand the inner workings of the heap and stack.

At what point do I have to worry about filling up the buffers in the network library and/or the wiz? (I'm also using the audio board and ethernetAudio so I'm conscious of making sure I don't hang the code up for too long. )
I see the Wiz has a 32kb for send/recv. One could only hope/assume that if it fills up it's buffers, that it will slow the TCP ACKs down?

Thanks!
 
Use Arduino's built-in String object. It will take care of the details for you. If you want to turn it into a normal ASCIIZ string, use String::c_str().
 
Use Arduino's built-in String object. It will take care of the details for you. If you want to turn it into a normal ASCIIZ string, use String::c_str().

From what I read in researching the issue, I've stayed away from using Strings and have preferred to keep my string manipulation "in house" using char arrays..

Since I am using libraries I did not write (so uncertain of their dynamic memory allocation use), as well as the fact that due to the nature of the project, I should never be handling more than one POST at a time, I have created a single global 20k buffer that will be used to store the POST payload. Seems wasteful..but as long as I don't run out of total space, I should be OK. With that instantiated, I'm at 23% mem usage. This project is intended to be in operation in remote locations and will be expected to run for for months/years without crash/reboot so proper memory handling is paramount. Sounds like without knowing the inner workings of the String class, I think I'd be better off sticking to char arrays.
 
Arduino devs often stay away from String because it allocates and frees memory. Over time, this can cause memory fragmentation where the largest available contiguous block shrinks over time, and eventually (after days to years) code that was otherwise fine will crash. So, if you want this to run reliably for years without reboot, I think it's a good idea to allocate statically as much as possible, use the stack where that isn't possible, and use the heap as little as possible.

That said, be sure you're guarding that buffer carefully against overflows. You may not overrun it today, but maybe you do something six months down the line that would occasionally go past the end of the buffer.

Also see to security, especially if this is accessed over the public internet (don't rely on "no one will figure out the IP") - put in some authentication mechanism, and harden it. You may also wish to use some kind of watchdog to force the thing to reboot if it hangs.
 
Status
Not open for further replies.
Back
Top