Uploading large amounts of data via HTTP POST

paulfer

paulfer

Member
Hi All,
I have just ventured into the TEENSY world, coming from the STM32 world.

I have a sketch, that triggers a HIKVISION camera to send thru a photo.

All works well. What it can do on the TEENSY4.1 so far quite successfully is:

  1. Trigger the camera using the NativeEthernet port (ISAPI and DIGEST auth).
  2. Save the image data to the DMBuffer.
  3. Save the image to the USB host port.
This works fabulously.

However, I also want to upload the image stored in the DMBuffer via POST and I am unsuccessful. I need guidance.

Here are some of the pertinent declarations

Code: 
#include <MD5.h>
#include <SPI.h>
#include <USBHost_t36.h>
#include "TimeLib.h"
#include "NativeEthernet.h"

/*========================================*/
#define DEBUG 1
/*========================================*/


/*========the ETHERNET stuff================*/
byte mac[] = { 0xDE, 0xAD, 0xBE, 0x1F, 0xFD, 0x1D };
IPAddress ip(192, 168, 1, 177);
IPAddress myDns(192, 168, 1, 1);
EthernetClient clientCAM;
EthernetClient clientLOGWIZ;
char serverCAM[] = "192.168.1.64";
//char serverLOGWIZ[] = "lexxatrexxa.co.za";
byte serverLOGWIZ[] = { 205, 144, 171, 137 };
const uint32_t postingInterval = 15 * 1000;  // delay between updates, in milliseconds
uint32_t ctr = 0;

/*========the USB stuff================*/

USBHost myusb;
USBHub hub1(myusb);
USBDrive mbDrive1(myusb);
USBFilesystem usbFS(myusb);

/*===========Variables=============*/
#define BUFSIZE 2048
#define BIGBUFSIZE 400000
#define CHUNKSIZE 2048  //65536  // 64KB  xferBuf size
#define REALMSIZE 32
#define NONCESIZE 65



bool fullres = true;
bool unauth = false;
bool isEthProblem = false;
bool isUSBProblem = false;
uint8_t attemptNumber = 1;
char cnonce[] = "MGZmYzRjNjU2YTM4NTUwZTJjOWY1YjdkMDM0Y2E2ODU=";  //generic cnonce
char md5ha2[33];
char realm[REALMSIZE];
char nonce[NONCESIZE];
uint8_t buf[BUFSIZE];
uint8_t DMBuffer[BIGBUFSIZE] DMAMEM;
uint8_t xferBuf[CHUNKSIZE];  //FASTRUN
/*========Timers==========*/
uint32_t timer1second = 0;
uint32_t lastConnectionTime = 0;


and this section where I read and save the image works fine too:

Code: 
/*==============2nd Attempt/DIGEST OK====================*/
    if (attemptNumber == 2) {

#if DEBUG == 1
      Serial.println("2:");
      uint32_t x = millis();
#endif
      /* clear the DMA buffer */
      memset(DMBuffer, 0, BIGBUFSIZE);
      uint32_t bxctr = 0;
      delay(10);

      /*  create thr USB filename and file */
      File testFile;
      char filename[50];
      memset(filename, 0, 50);
      sprintf(filename, "%s_%04i_%02i_%02i_%02i_%02i_%02i_199.lwz", "CA", year(), month(), day(), hour(), minute(), second());

      if (usbFS.exists(filename)) usbFS.remove(filename);
      testFile = usbFS.open(filename, FILE_WRITE);

      /*  add the filename to the body of the USB file  */
      int filenamelength = strlen(filename);
      testFile.write(&filename[0], filenamelength);

      /*  add the filename to the body of the DMABuffer   */
      for (int i = 0; i < filenamelength; i++) {
        DMBuffer[bxctr] = filename[i];
        bxctr++;
      }
      /*  read all the data from the camera in chucks of 2048  */
      while (clientCAM.available()) {
        memset(buf, 0, BUFSIZE);
        int bx = clientCAM.read(buf, BUFSIZE - 1);
        for (int i = 0; i < bx; i++) {
          DMBuffer[bxctr] = buf[i];
          bxctr++;
        }
        /*  append the data to the USB file if it exists  */
        if (testFile) {
          testFile.write(&buf[0], bx);
        }
        delay(1);
      }

      testFile.close();
      myusb.Task();


BUT immediatley after that I try to upload. And no matter how I upload, success is never achieved.

I tried simply posting it, but I understand that the buffer is only 2048 bytes big.

So this does not go:
Code: 
if (connectToLogwiz()) {
    
        clientLOGWIZ.println("POST /cw3/cwpost.php HTTP/1.1");
        clientLOGWIZ.println("Host: lexxatrexxa.co.za");
        clientLOGWIZ.println("Content-Type: application/octet-stream");
        clientLOGWIZ.println("Connection: keep-alive");
        clientLOGWIZ.print("Content-Length: ");
        clientLOGWIZ.println(bxctr);
        clientLOGWIZ.println();
  
        for (uint32_t i = 0; i < bxctr; i++) {
          clientLOGWIZ.write(DMBuffer[i]);     
        }

I tried breaking it up into 2048 byte chunks, but that too, did not work.

Even writing in 200 byte chunks does not work:

Code: 
char tempbuf[200];
        for (uint32_t i = 0; i < bxctr; i += 200) {
          memset(tempbuf, 0, 200);
          strncpy(tempbuf, &DMBuffer[i], 200);
          int buflen = strlen(tempbuf);
          clientLOGWIZ.write(&DMBuffer[i], buflen);
          //clientLOGWIZ.flush();//with or without flush yields same failure
        }

I then tried only uploading the first 200 bytes of DMBuffer and that DOES WORK.

So my only question is how do I get to POST a 245KB image to the server?

here is the connectToLogwiz() code:

Code: 
bool connectToLogwiz() {
  clientLOGWIZ.stop();
  if (clientLOGWIZ.connect(serverLOGWIZ, 80)) {
#if DEBUG == 1
    Serial.println("Logwiz OK");
#endif
    return true;
  } else {
#if DEBUG == 1
    Serial.println("Logwiz FAIL");
#endif
    return false;
  }
}
 
Last edited:
Recommend you use QNEthernet rather than NativeEthernet, which is no longer maintained. QNEthernet is developed and maintained by @shawn, a frequent contributor here.
 
Yes, it is designed to generally work as a direct replacement. Check the examples. One diff is that it even knows to get the Teensy direct MAC address.
 
Just some feedback,

I used the following code with the QNEthernet library
Code: 
char url[300];
        sprintf(url,
                "POST /xxx.php HTTP/1.1\r\nHost: lexxatrexxa.co.za\r\nontent-Type: application/octet-stream\r\nConnection: keep-alive\r\nContent-Length: %lu\r\n\r\n", bxctr);
        clientLOGWIZ.writeFully(url);
        clientLOGWIZ.writeFully(DMBuffer, bxctr);
        clientLOGWIZ.flush();

and the upload works perfectly, as well as only taking 2.5 seconds to take the picture, save to USB and upload to server! All that on my old African internet line!

Thanks again. How do I mark this as solved?
 
paulfer said:
And just to be sure, the QNEthernet works on the Native Ethernet port of the Teensy4.1?
Click to expand...

Just to disambiguate some terms: The Teensy 4.1 has support for “native Ethernet”, meaning there’s a PHY chip on-board for interfacing with Ethernet ports. There’s also a library called “NativeEthernet” that happens to be included with the Teensyduino distribution. It was likely included because it was the first library to include support for the Teensy’s on-board “native” hardware.
 
shawn said:
Just to disambiguate some terms: The Teensy 4.1 has support for “native Ethernet”, meaning there’s a PHY chip on-board for interfacing with Ethernet ports. There’s also a library called “NativeEthernet” that happens to be included with the Teensyduino distribution. It was likely included because it was the first library to include support for the Teensy’s on-board “native” hardware.
Click to expand...
Thanks Shawn. I used your QNEthernet and the Ethernet kit from PJRC, and it works a treat! I am gobsmacked at the speeds I am getting. Revolutionised my project TBH.
 
You must log in or register to reply here.
Back
Top