T4.1 NativeEthernet library question (WebServer/process a POST)

jahwerx

Member
Hi all,

Recently migrating a project from an ESP to Teensy 4.1.
My old code used ESP8266WebServer.h and now I'm moving over to NativeEthernet.

Here's what I've been able to successfully do so far:
  • Get the Teensy on my network
  • Enable mDNS and a basic webserver
  • Get a client to connect; The Teensy is serving up my HTML/CSS/jScript and it renders great.
  • Looking at the serial output, the POST is coming through!

What isn't really clear is how to process the data. I need the syntax to be able to get to my form data. I've been using the googles to try to find some documentation on the library, and have come up empty handed. In the ESP library, it was as simple as taking server.arg(0) (First form element value), server.arg(1) (Second Form element value), etc.

So effectively I've got a form with a bunch UX elements that I want the user to interact with and when they "post" have the Teensy do it's thing (in this case, control a bunch of LEDs). I've also gotten the Teensy able to drive the LED's with different patterns, etc, so I got that part working as well... I just need to marry these together.

If anyone has an example of handling a POST in the webserver and managing the form elements, I can take it from here.

Thanks in advance,
J
 
You might be looking for the “application/x-www-form-urlencoded” content type. Or perhaps even “multipart/form-data”. See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST
Shawn, Firstly, thank you. I thought maybe forcing the form to text/plain might solve my problem.
However, no matter what encoding method I try, It's not entirely obvious on how the POST data is getting brought back... I'm SURE it's coming back; but using client.read is clearly not the right method... This is where I'm struggling. I know someone must have the solution :)

So I've pretty much given up with POST at this point.
What I've been able to get working is using GET, although I really hate this as it just bangs everything in the URL... but at least I can parse this out now.

Here's the response I get with "GET":
Code:
new client
GET /payload?fname=Hi&lname=There HTTP/1.1 <= YES, I can get my form data here!!
Host: daboosh.local
Connection: keep-alive
Upgrade-Insecure-Requests: 1
...

Here are the various POST tests - I feel the data is "just out of reach"
(multipart/form-data)
Code:
new client
POST /payload HTTP/1.1
Host: daboosh.local
Connection: keep-alive
Content-Length: 238 <= WHERE IS THE CONTENT?
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: http://daboosh.local
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryVrAPwAqlTfYBEvxE  <= WHERE IS THE CONTENT?
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
...

(x-www-form-urlencoded)
Code:
new client
POST /payload HTTP/1.1
Host: daboosh.local
Connection: keep-alive
Content-Length: 22 <= WHERE IS THE CONTENT?
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: http://daboosh.local
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
...

(Good old text/plain)
Code:
new client
POST /payload HTTP/1.1
Host: daboosh.local
Connection: keep-alive
Content-Length: 25 <= WHERE IS THE CONTENT?
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: http://daboosh.local
Content-Type: text/plain
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: http://daboosh.local/payload
...

And finally here is the test script (It's pretty much 100% "example WebServerMDNS" with the HTML swapped out for the form...)
Code:
#include "NativeEthernet.h"

uint8_t mac[6];
void teensyMAC(uint8_t *mac) {
    for(uint8_t by=0; by<2; by++) mac[by]=(HW_OCOTP_MAC1 >> ((1-by)*8)) & 0xFF;
    for(uint8_t by=0; by<4; by++) mac[by+2]=(HW_OCOTP_MAC0 >> ((3-by)*8)) & 0xFF;
    Serial.printf("MAC: %02x:%02x:%02x:%02x:%02x:%02x\n", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
}

//EthernetServer server(443, true); //Uncomment for TLS
EthernetServer server(80);

void setup() {;
  // put your setup code here, to run once:
  delay(3000);
  teensyMAC(mac);
  Ethernet.begin(mac);
  MDNS.begin("DaBoosh", 2); //.local Domain name and number of services
  MDNS.setServiceName("Teensy41_Service_Name"); //Uncomment to change service name
  MDNS.addService("_https._tcp", 443); //Uncomment for TLS
  MDNS.addService("_http._tcp", 80);
  server.begin();
  Serial.print("IP  address: ");
  Serial.println(Ethernet.localIP());
  Serial.send_now();
}

void loop() {
  // listen for incoming clients
  EthernetClient client = server.available();
  if (client) {
    Serial.println("new client");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println("Connection: close");  // the connection will be closed after completion of the response
          // client.println("Refresh: 5");  // refresh the page automatically every 5 sec
          client.println();

          // Here is the basic form information
          client.println("<!DOCTYPE html>\n");
          client.println("<html>\n");
          client.println("<body>\n");
          client.println("<h1>Testing the Forms</h1>\n");
          client.println("<p>Try a POST and GET</p>\n");
          client.println("<form action=\"/payload\" method=\"get\" target=\"_blank\">\n");
          client.println("  <label for=\"fname\">First name:</label>\n");
          client.println("  <input type=\"text\" id=\"fname\" name=\"fname\"><br><br>\n");
          client.println("  <label for=\"lname\">Last name:</label>\n");
          client.println("  <input type=\"text\" id=\"lname\" name=\"lname\"><br><br>\n");
          client.println("  <input type=\"submit\" value=\"Submit using GET\">\n");
          client.println("  <input type=\"submit\" formmethod=\"post\" formenctype=\"multipart/form-data\" value=\"Submit POST as Multipart/form-data\">\n");
          client.println("  <input type=\"submit\" formmethod=\"post\" formenctype=\"application/x-www-form-urlencoded\" value=\"Submit POST as x-www-form-urlencoded\">\n");
          client.println("  <input type=\"submit\" formmethod=\"post\" formenctype=\"text/plain\" value=\"Submit POST as text/plain\">");
          client.println("  <input type=\"submit\" formmethod=\"post\" value=\"Submit using POST\">\n");
          client.println("</form>\n");
          client.println("</body>\n");
          client.println("</html>");
         
          delay(500);
          client.close(); //If "Connection: close" make sure to close after printing and before stop to avoid harsh reset
          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(500);
    // close the connection:
    client.stop();
    Serial.println("client disconnected");
  }
}
 
Oh my goodness... I figured it out.

The sample code makes the assumption that the client response is complete with a new line and a blank line, and was breaking before all the data could be written out.

Simply by removing the break I'm able to see the data now :)

Code:
new client
POST /payload HTTP/1.1
Host: daboosh.local
Connection: keep-alive
Content-Length: 23
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: http://daboosh.local
Content-Type: text/plain
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: http://daboosh.local/payload
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9

fname=Hello&lname=World <=TADA. There it is!!! Content length is indeed 23 bytes
client disconnected

Hopefully this can help others out, and now I can code all this up properly.
 
Note: `println` doesn't necessarily guarantee that all the bytes are written. Technically, you need to loop until all the bytes are written. Additionally, the `print` and `println` calls don't always accurately return the number of bytes transmitted, depending on which version you're calling from which library. The best way to send data like this is actually using `write` calls. QNEthernet provides a `writeFully()` function that does this for you. (See https://forum.pjrc.com/threads/6838...ropped-packets?p=290568&viewfull=1#post290568 for code, or copy the code from QNEthernetClient.cpp from QNEthernet.)

Some additional notes:
1. If you do choose to use the `writeFully()` approach instead, make sure to add the "\r\n" bytes where you want line breaks.
2. The above point only applies to lines that need it, for example, HTTP headers. HTML doesn't actually need these. Their utility comes when you want to look at the page code and have it formatted a little nicer, rather than it being on one line. If formatting isn't important to you, you don't need the CRLF bytes.
3. I would add a `flush()` call after all your data is fully written. Using some delay isn't a guarantee that all the data has actually gone out, unless the delay time is sufficiently long, but that depends on a lot of factors. 500ms is probably enough, but it's uncertain.

By "removing the break," did you mean that you removed the `break;` line after the `client.close()` call?
 
Last edited:
Note: `println` doesn't necessarily guarantee that all the bytes are written. Technically, you need to loop until all the bytes are written. Additionally, the `print` and `println` calls don't always accurately return the number of bytes transmitted, depending on which version you're calling from which library. The best way to send data like this is actually using `write` calls. QNEthernet provides a `writeFully()` function that does this for you. (See https://forum.pjrc.com/threads/6838...ropped-packets?p=290568&viewfull=1#post290568 for code, or copy the code from QNEthernetClient.cpp from QNEthernet.)

Some additional notes:
1. If you do choose to use the `writeFully()` approach instead, make sure to add the "\r\n" bytes where you want line breaks.
2. The above point only applies to lines that need it, for example, HTTP headers. HTML doesn't actually need these. Their utility comes when you want to look at the page code and have it formatted a little nicer, rather than it being on one line. If formatting isn't important to you, you don't need the CRLF bytes.
3. I would add a `flush()` call after all your data is fully written. Using some delay isn't a guarantee that all the data has actually gone out, unless the delay time is sufficiently long, but that depends on a lot of factors. 500ms is probably enough, but it's uncertain.

By "removing the break," did you mean that you removed the `break;` line after the `client.close()` call?

Thank you very much for the tips on QNEthernet.
Removing the break after client.close: EXACTLY correct; I just commented the line. It allowed the form data to come through... which happens AFTER a blank line/new line. (Note the blank line after "Accept-Language: en-US,en;q=0.9"). I'm surprised I didn't spot this earlier.

So I am currently able to grab, parse, and assign the form data to variables perfectly. HOWEVER, I now have another issue. fastLED processing breaks when I invoke Ethernet.begin :( but I'll start a new thread for that. This has been a bit of a struggle migrating from ESP...
 
Back
Top