Forum Rule: Always post complete source code & details to reproduce any issue!
Page 1 of 2 1 2 LastLast
Results 1 to 25 of 29

Thread: 400 Bad Request while posting to InfluxDB with NativeEthernet

  1. #1

    400 Bad Request while posting to InfluxDB with NativeEthernet

    Hi,

    For a project we are trying to monitor a status protocol sent on local network over TCP and post these status messages over HTTPS to a InfluxDB instance on AWS at https://www.influxdata.com/products/influxdb-cloud/.

    We are using a teensy 4.1 with a ethernet kit in combination with the Native Ethernet library by vjmuzik https://github.com/vjmuzik/NativeEthernet.

    Posting a string to a test instance over LAN to a IP works fine, but whenever we try to post to the AWS subdomain we are greeted with this response:
    HTTP/1.1 400 Bad Request
    Server: awselb/2.0
    Date: Mon, 27 Sep 2021 18:13:02 GMT
    Content-Type: text/html
    Content-Length: 122
    Connection: close

    <html>
    <head><title>400 Bad Request</title></head>
    <body>
    <center><h1>400 Bad Request</h1></center>
    </body>
    </html>

    Since we verified the headers and body of the post are correct when we post locally, and posting to AWS works when using Postman, we are at a loss at the moment. Unfortunately I have no information about the configuration of the server we try to POST to.

    As a test we set-up the WebClientRepeatingTLS example using our own POST headers and body.

    Code:
    #include <NativeEthernet.h>
    
    uint8_t mac[6];
    void teensyMAC(uint8_t *mac) {
      for (uint8_t by = 0; by < 2; by++) mac[by] = (HW_OCOTP_MAC1 >> ((1 - by) * 8)) & 0xFF;
      for (uint8_t by = 0; by < 4; by++) mac[by + 2] = (HW_OCOTP_MAC0 >> ((3 - by) * 8)) & 0xFF;
      Serial.printf("MAC: %02x:%02x:%02x:%02x:%02x:%02x\n", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
    }
    
    // initialize the library instance:
    EthernetClient client;
    const int port = 443;
    
    char server[] = "influxdata.com";  
    
    //IPAddress server(192, 168, 1, 246);
    
    String outputMSG = "measurementName,tagKey=tagValue fieldKey=1i";
    unsigned long lastConnectionTime = 0;           // last time you connected to the server, in milliseconds
    const unsigned long postingInterval = 5 * 1000; // delay between updates, in milliseconds
    
    void setup() {
      teensyMAC(mac);
      Serial.begin(9600);
      while (!Serial) {
        ; 
      }
    
      Serial.println("Initialize Ethernet with DHCP:");
      if (Ethernet.begin(mac) == 0) {
        Serial.println("Failed to configure Ethernet using DHCP");
        if (Ethernet.hardwareStatus() == EthernetNoHardware) {
          Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
          while (true) {
            delay(1);
          }
        }
        if (Ethernet.linkStatus() == LinkOFF) {
          Serial.println("Ethernet cable is not connected.");
        }
      } else {
        Serial.print("  DHCP assigned IP ");
        Serial.println(Ethernet.localIP());
      }
      delay(1000);
    }
    
    void loop() {
      if (client.available()) {
        char c = client.read();
        Serial.write(c);
      }
      if (millis() - lastConnectionTime > postingInterval) {
        httpRequest();
      }
    
    }
    
    void httpRequest() {
      client.stop();
      Serial.print("Content-Length: ");
      Serial.println(outputMSG.length());
      Serial.println("");
    
      if (client.connect(server, port, true)) { 
        Serial.println("connecting to ");
        Serial.print(server);
        Serial.print(":");
        Serial.println(port);
    
            client.println("POST /api/v2/write?org=ORG_HERE&bucket=BUCKET_HERE&precision=s HTTP/1.1");
            client.println("Host: https://eu-central-1-1.aws.cloud2.influxdata.com");
            client.println("Authorization: Token <<TOKEN HERE>>");
            client.println("Content-Type: text/plain");
            client.print("Content-Length: ");
            client.println(outputMSG.length());
            client.println("Accept-Encoding: gzip, deflate, br");
            client.println("Connection: keep-alive");
            client.println();
            client.println(outputMSG);
            client.println();
            
        lastConnectionTime = millis();
      } else {
        Serial.println("connection failed");
      }
    }
    Could anyone help us understand why the post to aws is not going through?


    Cheers,
    Boy

  2. #2
    Senior Member
    Join Date
    Jul 2020
    Posts
    1,364
    Quote Originally Posted by bvanpoortvliet View Post
    Code:
            client.println("POST /api/v2/write?org=ORG_HERE&bucket=BUCKET_HERE&precision=s HTTP/1.1");
            client.println("Host: https://eu-central-1-1.aws.cloud2.influxdata.com");
            client.println("Authorization: Token <<TOKEN HERE>>");
            client.println("Content-Type: text/plain");
            client.print("Content-Length: ");
            client.println(outputMSG.length());
            client.println("Accept-Encoding: gzip, deflate, br");
            client.println("Connection: keep-alive");
            client.println();
            client.println(outputMSG);
            client.println();
    Could anyone help us understand why the post to aws is not going through?


    Cheers,
    Boy
    Perhaps that's not generating valid HTTP headers. Every line in an HTTP header _must_ end with \r\n, including the blank line at the end of the headers.
    I don't know if the EthernetClient library does this for println (this restriction only applies to HTTP headers, not body content).

  3. #3
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    349
    Couple things:
    1. Since this is an HTTP/1.1 request, and I’m willing to bet the endpoint is at least an HTTP/1.1 server, I’d either use “Connection: close” (to close the connection right after) or remove that header (to keep it alive).
    2. You’re sending 4 extra bytes after outputMSG, I which, in the case of a connection that’s reused, will be interpreted as part of the next request. I.e. two extra “\r\n”. You’ve specified the content length to be the size of outputMSG.

  4. #4
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    349
    Additions:
    3. Teensy’s `println` functions append “\r\n”. The Arduino reference says that also for a few of the `println` definitions.
    4. Some servers accept empty lines before a request, but there’s no guarantee.

  5. #5
    Senior Member
    Join Date
    Jul 2020
    Posts
    1,364
    OK its not line endings then, but taking another look this is a worry:
    Code:
    client.println("Host: https://eu-central-1-1.aws.cloud2.influxdata.com");
    Since the Host line takes a domain and port number, not a URL, ie "Host: eu-central-1-1.aws.cloud2.influxdata.com:443"

  6. #6
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    349
    Good catch. According to RFC 7230, the host starts with uri-host, defined here: https://datatracker.ietf.org/doc/htm...#section-3.2.2

    Update: It’s possible RFC 2616 allowed a scheme in the Host. After a quick glance, I’m not actually certain.
    Last edited by shawn; 10-01-2021 at 03:16 AM.

  7. #7
    Senior Member vjmuzik's Avatar
    Join Date
    Apr 2017
    Posts
    833
    Shouldn’t the server address match the subdomain address from the host line? Likely DNS isn’t getting the IP address to the subdomain, so it would be reporting as a bad request since it’s routing to the main website.

  8. #8
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    349
    The Host value doesn’t necessarily need to identically match the server address. Also, if the server did have a problem with it, a 400 response doesn’t seem appropriate to me.

  9. #9
    Senior Member vjmuzik's Avatar
    Join Date
    Apr 2017
    Posts
    833
    If it’s not being sent to the IP Address of the subdomain, will sending to the main domain automatically reroute it to the subdomain in the host value? I wouldn’t really expect it to myself, but I don’t know enough about HTTP or AWS.

  10. #10
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    349
    Where the client is routed for a given Host is up to the server. Its intent is for “multihoming”. See: https://datatracker.ietf.org/doc/htm...appendix-A.1.1

    My guess is that that 400 error came from the scheme in the Host header. @MarkT, did fixing that solve the problem?

  11. #11
    Senior Member vjmuzik's Avatar
    Join Date
    Apr 2017
    Posts
    833
    I can see how that would support multihoming, but that also assumes the subdomain resolves to the same IP Address as the main domain when doing a DNS lookup. That may or may not be the case, so the DNS lookup should still be done for the subdomain and not the main domain.

  12. #12
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    349
    The contents of the Host field doesn't necessarily need a DNS lookup, and doesn't have to be related to the server address you've connected to. (If that's what you mean.)

  13. #13
    Senior Member vjmuzik's Avatar
    Join Date
    Apr 2017
    Posts
    833
    If the subdomain is being hosted on a different server than the main domain, which is likely the case, the client socket would need to be connected to that server which more than likely has a different IP Address. Currently the DNS lookup is for the main domain which could result in a different IP Address than the subdomain therefore it’s not connected to the right place. Being that he said it worked locally over LAN to an IP Address I suspect this to be the issue.

  14. #14
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    349
    I mean, that could be what’s happening on the server, sure, but in the general case, the Host value doesn’t need to match. (I’m commenting on the general case, I realize, and your answer may be more closely addressing the problem.)

    I guess it depends on the server API. You could be right here. @bvanpoortvliet, do you have more information about the server requirements?

  15. #15
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    349
    I think examples will help. I’m saying: you could connect to serverx.com and put “trees.com” in the Host header. Then the server will use the “trees.com” content (whatever it has and decides internally) to fulfill the request.

    @vjmuzik: are you saying that the server influxdata.com is seeing “eu-central-1-1.aws.cloud2.influxdata.com” in the Host header and giving a Bad Request because it doesn’t know how to handle requests for “eu-central-1-1.aws.cloud2.influxdata.com”?
    Last edited by shawn; 10-01-2021 at 10:42 AM.

  16. #16
    Senior Member vjmuzik's Avatar
    Join Date
    Apr 2017
    Posts
    833
    That’s exactly what I’m saying because they are hosted by different servers, a quick ping from my phone confirms this, then the main domain has no idea about the content on the subdomain. If it was hosted by the same machine, same IP Address, then it would know how to handle the request.

  17. #17
    Wow, thanks so much for the replies! This was a real educational read so far. Also, pardon my late response, I had a bit of setting up to do. I have tried to run some tests with your given info, hoping I understand everything correctly.

    I edited the sketch to post the right data to my local influxdb server, which responded with the expected:
    Code:
    HTTP/1.1 204 No Content
    Date: Sat, 02 Oct 2021 12:13:42 GMT
    Connection: close

    Which means the post to the database was successfull, as confirmed by the database itself:
    Code:
    table 	_start 							_stop 				_time 			_value	 _field 	 _measurement		tagKey
    0 		2021-10-02T11:14:44.488Z 2021-10-02T	12:14:44.488Z 2021-10-02	T12:14:40.000Z 	1 		fieldKey	 measurementName 	tagValue

    Then I changed the port in the sketch to the port of my webserver so I could log the request.

    Code:
    [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 71 bytes
    dumpio_in (data-HEAP): POST /api/v2/write?org=40c706b395812822&precision=s&bucket=boy HTTP/1.1
    dumpio_in (data-HEAP): 2 bytes
    dumpio_in (data-HEAP): \r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 21 bytes
    dumpio_in (data-HEAP): Host: 192.168.1.145\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 111 bytes
    dumpio_in (data-HEAP): Authorization: Token <<NOTSHOWINGTOKEN>>\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 26 bytes
    dumpio_in (data-HEAP): Content-Type: text/plain\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 25 bytes
    dumpio_in (data-HEAP): User-Agent: Arduino/1.0\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 20 bytes
    dumpio_in (data-HEAP): Content-Length: 43\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 19 bytes
    dumpio_in (data-HEAP): Connection: close\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 2 bytes
    dumpio_in (data-HEAP): \r\n
    dumpio_in [readbytes-blocking] 43 readbytes
    dumpio_in (data-HEAP): 43 bytes
    dumpio_in (data-HEAP): measurementName,tagKey=tagValue fieldKey=1i
    dumpio_out
    dumpio_out (data-HEAP): 180 bytes
    Then I made a request with postman to the webserver so we can compare the requests.
    Code:
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 73 bytes
    dumpio_in (data-HEAP): POST /api/v2/write?org=40c706b395812822&precision=s&bucket=boy HTTP/1.1\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 111 bytes
    dumpio_in (data-HEAP): Authorization: Token <<DIFFERENTTOKENSTILLNOTSHOWINGIT>>\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 26 bytes
    dumpio_in (data-HEAP): Content-Type: text/plain\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 35 bytes
    dumpio_in (data-HEAP): User-Agent: PostmanRuntime/7.28.4\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 13 bytes
    dumpio_in (data-HEAP): Accept: */*\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 53 bytes
    dumpio_in (data-HEAP): Postman-Token: <<NOPE>>\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 21 bytes
    dumpio_in (data-HEAP): Host: 192.168.1.245\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 36 bytes
    dumpio_in (data-HEAP): Accept-Encoding: gzip, deflate, br\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 24 bytes
    dumpio_in (data-HEAP): Connection: keep-alive\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 20 bytes
    dumpio_in (data-HEAP): Content-Length: 43\r\n
    dumpio_in [getline-blocking] 0 readbytes
    dumpio_in (data-HEAP): 2 bytes
    dumpio_in (data-HEAP): \r\n
    dumpio_in [readbytes-blocking] 43 readbytes
    dumpio_in (data-HEAP): 43 bytes
    dumpio_in (data-HEAP): measurementName,tagKey=tagValue fieldKey=1i
    dumpio_out
    Couple of things I found interresting; first, the line end in the influxdb POST from the teensy seems to be sent separate from the post line in comparison to Postman. Second, it seems that the body does not have \r\n, but since the body does get accepted by the database I'm not worried about this.

    In order to be as complete as possible I have commented the empty println separating the headers from the body and re-posted to the database, this was the result:
    Code:
    HTTP/1.1 400 Bad Request
    Content-Type: text/plain; charset=utf-8
    Connection: close
    This shows that the empty println is necessary in the request.

    I have tried to remove https:// and add :443 to the Host when trying to post to AWS, which generated the response:

    Code:
    HTTP/1.1 301 Moved Permanently
    Server: awselb/2.0
    Date: Sat, 02 Oct 2021 12:42:08 GMT
    Content-Type: text/html
    Content-Length: 0
    Connection: close
    Location: https://www.influxdata.com/api/v2/write
    Unfortunately I do not have more info about the hosted part of the server as it is being taken care of by Influx itself. I do have some links to the line protocol and API for you.

    https://docs.influxdata.com/influxdb/v2.0/
    https://docs.influxdata.com/influxdb...tion/PostWrite
    https://docs.influxdata.com/influxdb...line-protocol/

    @shawn regarding the examples remark, is this what you mean?


    Code:
    char server[] = "google.com";  // also change the Host line in httpRequest()
    
    client.println("GET /index.html HTTP/1.1");
    client.println("Host: www.pjrc.com");
    client.println("User-Agent: arduino-ethernet");
    client.println("Connection: close");
    client.println();
    client.close(); //If "Connection: close" make sure to close after printing and before stop to avoid harsh reset
    I got this response:
    Code:
    connecting...
    HTTP/1.1 404 Not Found
    Content-Type: text/html; charset=UTF-8
    Referrer-Policy: no-referrer
    Content-Length: 1571
    Date: Sat, 02 Oct 2021 13:13:13 GMT
    Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
    Connection: close
    
    <!DOCTYPE html>
    <html lang=en>
      <meta charset=utf-8>
      <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
      <title>Error 404 (Not Found)!!1</title>
      <style>
        *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
      </style>
      <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
      <p><b>404.</b> <ins>That’s an error.</ins>
      <p>The requested URL <code>/index.html</code> was not found on this server.  <ins>That’s all we know.</ins>

  18. #18
    Junior Member
    Join Date
    Mar 2021
    Posts
    16
    What if you do not use client.println but client.print and put the \r\n at the end of the string:

    client.print("POST /api/v2/write?org=ORG_HERE&bucket=BUCKET_HERE&precision=s HTTP/1.1\r\n");

  19. #19
    Senior Member vjmuzik's Avatar
    Join Date
    Apr 2017
    Posts
    833
    Quote Originally Posted by RoSchmi View Post
    What if you do not use client.println but client.print and put the \r\n at the end of the string:

    client.print("POST /api/v2/write?org=ORG_HERE&bucket=BUCKET_HERE&precision=s HTTP/1.1\r\n");
    Realistically speaking, even though it doesn’t arrive in the same packet that does not cause issues for TCP and he says it was accepted locally so there’s nothing wrong with the formatting of his message. I still maintain that this is purely because it’s being sent to the wrong IP Address due to the fact that he hasn’t included the subdomain in the server char array.

  20. #20
    @vjmuzik, I agree. However, when I try to include the subdomain in the server char array the connection fails.

  21. #21
    I just realized I have a spare ESP board laying around, so I decided to give Influxdb client for Arduino a try to exclude network issues on my end InfluxDB-Client-for-Arduino.
    This library worked like a charm out of the box. Unfortunately I am not in the position to make use of wireless networking for this project.

    A quick ping to eu-central-1-1.aws.cloud2.influxdata.com comes back with request timeout but shows me aed270f02d59711e9aaa902ee3ef9179-9aec589b0bda0555.elb.eu-central-1.amazonaws.com as an address. This combined with the server response header in my original post awselb/2.0 makes me believe I am hitting the Elastic Loadbalancer in AWS. What I don't understand is why the connection stops there and does not get forwarded.

    @RoSchmi did you have any luck working with TLS rootCa verification after your git post? I can imagine this could be an issue as well since the ESP library uses BearSSL with certificates and my sketch does not.
    Last edited by bvanpoortvliet; 10-03-2021 at 11:43 AM.

  22. #22
    Junior Member
    Join Date
    Mar 2021
    Posts
    16
    Quote Originally Posted by bvanpoortvliet View Post

    @RoSchmi did you have any luck working with TLS rootCa verification after your git post? I can imagine this could be an issue as well since the ESP library uses BearSSL with certificates and my sketch does not.
    I'm sorry but I don't exactly understand what you mean, please describe a little bit more to what you are exactly referring.
    On my Github account 'RoSchmi' I have some projects for different platforms (e.g. Teensy, Esp32, Wio Terminal) storing sensor data to Azure Storage Tables. For the Teensy e.g. -https://github.com/RoSchmi/AzureDataSender_Teensy.
    Do you mean this? The Teensy example works with bear_ssl and the http client from -https://github.com/khoih-prog/EthernetWebServer_SSL

  23. #23
    Sorry, that was a very vague description on my end. I am referring to this issue on vjmuzik's NativeEthernet library git and was wondering if you got rootCa verification working with the NativeEthernet library. Regarding the EthernetWebServer_SSL library, I have tried to use my sketch with this library as well but got the same results as described in my initial post.

  24. #24
    Senior Member vjmuzik's Avatar
    Join Date
    Apr 2017
    Posts
    833
    Make sure you update FNET to the latest version, there was a problem with DNS not resolving aliases and it sounds like that may be the issue now.

  25. #25
    Both NativeEthernet and FNET are the latest version. I just did a test with a teensy 3.2 in combination with the EthernetWebServer_SSL and it works as expected when USE_ETHERNET is set to true in defines, so it uses Ethernet.h. Could it be that there's still some issues with DNS resolving in NativeEthernet?

    I also checked if can post to influxdb with EthernetWebServer_SSL and Ethernet3 on my teensy 3.2, this functions properly as well.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •