400 Bad Request while posting to InfluxDB with NativeEthernet

Status
Not open for further replies.
Hi,

For a project we are trying to monitor a status protocol sent on local network over TCP and post these status messages over HTTPS to a InfluxDB instance on AWS at https://www.influxdata.com/products/influxdb-cloud/.

We are using a teensy 4.1 with a ethernet kit in combination with the Native Ethernet library by vjmuzik https://github.com/vjmuzik/NativeEthernet.

Posting a string to a test instance over LAN to a IP works fine, but whenever we try to post to the AWS subdomain we are greeted with this response:
HTTP/1.1 400 Bad Request
Server: awselb/2.0
Date: Mon, 27 Sep 2021 18:13:02 GMT
Content-Type: text/html
Content-Length: 122
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>

Since we verified the headers and body of the post are correct when we post locally, and posting to AWS works when using Postman, we are at a loss at the moment. Unfortunately I have no information about the configuration of the server we try to POST to.

As a test we set-up the WebClientRepeatingTLS example using our own POST headers and body.

Code:
#include <NativeEthernet.h>

uint8_t mac[6];
void teensyMAC(uint8_t *mac) {
  for (uint8_t by = 0; by < 2; by++) mac[by] = (HW_OCOTP_MAC1 >> ((1 - by) * 8)) & 0xFF;
  for (uint8_t by = 0; by < 4; by++) mac[by + 2] = (HW_OCOTP_MAC0 >> ((3 - by) * 8)) & 0xFF;
  Serial.printf("MAC: %02x:%02x:%02x:%02x:%02x:%02x\n", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
}

// initialize the library instance:
EthernetClient client;
const int port = 443;

char server[] = "influxdata.com";  

//IPAddress server(192, 168, 1, 246);

String outputMSG = "measurementName,tagKey=tagValue fieldKey=1i";
unsigned long lastConnectionTime = 0;           // last time you connected to the server, in milliseconds
const unsigned long postingInterval = 5 * 1000; // delay between updates, in milliseconds

void setup() {
  teensyMAC(mac);
  Serial.begin(9600);
  while (!Serial) {
    ; 
  }

  Serial.println("Initialize Ethernet with DHCP:");
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    if (Ethernet.hardwareStatus() == EthernetNoHardware) {
      Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
      while (true) {
        delay(1);
      }
    }
    if (Ethernet.linkStatus() == LinkOFF) {
      Serial.println("Ethernet cable is not connected.");
    }
  } else {
    Serial.print("  DHCP assigned IP ");
    Serial.println(Ethernet.localIP());
  }
  delay(1000);
}

void loop() {
  if (client.available()) {
    char c = client.read();
    Serial.write(c);
  }
  if (millis() - lastConnectionTime > postingInterval) {
    httpRequest();
  }

}

void httpRequest() {
  client.stop();
  Serial.print("Content-Length: ");
  Serial.println(outputMSG.length());
  Serial.println("");

  if (client.connect(server, port, true)) { 
    Serial.println("connecting to ");
    Serial.print(server);
    Serial.print(":");
    Serial.println(port);

        client.println("POST /api/v2/write?org=ORG_HERE&bucket=BUCKET_HERE&precision=s HTTP/1.1");
        client.println("Host: https://eu-central-1-1.aws.cloud2.influxdata.com");
        client.println("Authorization: Token <<TOKEN HERE>>");
        client.println("Content-Type: text/plain");
        client.print("Content-Length: ");
        client.println(outputMSG.length());
        client.println("Accept-Encoding: gzip, deflate, br");
        client.println("Connection: keep-alive");
        client.println();
        client.println(outputMSG);
        client.println();
        
    lastConnectionTime = millis();
  } else {
    Serial.println("connection failed");
  }
}

Could anyone help us understand why the post to aws is not going through?


Cheers,
Boy
 
Code:
        client.println("POST /api/v2/write?org=ORG_HERE&bucket=BUCKET_HERE&precision=s HTTP/1.1");
        client.println("Host: https://eu-central-1-1.aws.cloud2.influxdata.com");
        client.println("Authorization: Token <<TOKEN HERE>>");
        client.println("Content-Type: text/plain");
        client.print("Content-Length: ");
        client.println(outputMSG.length());
        client.println("Accept-Encoding: gzip, deflate, br");
        client.println("Connection: keep-alive");
        client.println();
        client.println(outputMSG);
        client.println();

Could anyone help us understand why the post to aws is not going through?


Cheers,
Boy
Perhaps that's not generating valid HTTP headers. Every line in an HTTP header _must_ end with \r\n, including the blank line at the end of the headers.
I don't know if the EthernetClient library does this for println (this restriction only applies to HTTP headers, not body content).
 
Couple things:
1. Since this is an HTTP/1.1 request, and I’m willing to bet the endpoint is at least an HTTP/1.1 server, I’d either use “Connection: close” (to close the connection right after) or remove that header (to keep it alive).
2. You’re sending 4 extra bytes after outputMSG, I which, in the case of a connection that’s reused, will be interpreted as part of the next request. I.e. two extra “\r\n”. You’ve specified the content length to be the size of outputMSG.
 
Additions:
3. Teensy’s `println` functions append “\r\n”. The Arduino reference says that also for a few of the `println` definitions.
4. Some servers accept empty lines before a request, but there’s no guarantee.
 
OK its not line endings then, but taking another look this is a worry:
Code:
client.println("Host: https://eu-central-1-1.aws.cloud2.influxdata.com");
Since the Host line takes a domain and port number, not a URL, ie "Host: eu-central-1-1.aws.cloud2.influxdata.com:443"
 
Shouldn’t the server address match the subdomain address from the host line? Likely DNS isn’t getting the IP address to the subdomain, so it would be reporting as a bad request since it’s routing to the main website.
 
The Host value doesn’t necessarily need to identically match the server address. Also, if the server did have a problem with it, a 400 response doesn’t seem appropriate to me.
 
If it’s not being sent to the IP Address of the subdomain, will sending to the main domain automatically reroute it to the subdomain in the host value? I wouldn’t really expect it to myself, but I don’t know enough about HTTP or AWS.
 
I can see how that would support multihoming, but that also assumes the subdomain resolves to the same IP Address as the main domain when doing a DNS lookup. That may or may not be the case, so the DNS lookup should still be done for the subdomain and not the main domain.
 
The contents of the Host field doesn't necessarily need a DNS lookup, and doesn't have to be related to the server address you've connected to. (If that's what you mean.)
 
If the subdomain is being hosted on a different server than the main domain, which is likely the case, the client socket would need to be connected to that server which more than likely has a different IP Address. Currently the DNS lookup is for the main domain which could result in a different IP Address than the subdomain therefore it’s not connected to the right place. Being that he said it worked locally over LAN to an IP Address I suspect this to be the issue.
 
I mean, that could be what’s happening on the server, sure, but in the general case, the Host value doesn’t need to match. (I’m commenting on the general case, I realize, and your answer may be more closely addressing the problem.)

I guess it depends on the server API. You could be right here. @bvanpoortvliet, do you have more information about the server requirements?
 
I think examples will help. I’m saying: you could connect to serverx.com and put “trees.com” in the Host header. Then the server will use the “trees.com” content (whatever it has and decides internally) to fulfill the request.

@vjmuzik: are you saying that the server influxdata.com is seeing “eu-central-1-1.aws.cloud2.influxdata.com” in the Host header and giving a Bad Request because it doesn’t know how to handle requests for “eu-central-1-1.aws.cloud2.influxdata.com”?
 
Last edited:
That’s exactly what I’m saying because they are hosted by different servers, a quick ping from my phone confirms this, then the main domain has no idea about the content on the subdomain. If it was hosted by the same machine, same IP Address, then it would know how to handle the request.
 
Wow, thanks so much for the replies! This was a real educational read so far. Also, pardon my late response, I had a bit of setting up to do. I have tried to run some tests with your given info, hoping I understand everything correctly.

I edited the sketch to post the right data to my local influxdb server, which responded with the expected:
Code:
HTTP/1.1 204 No Content
Date: Sat, 02 Oct 2021 12:13:42 GMT
Connection: close


Which means the post to the database was successfull, as confirmed by the database itself:
Code:
table 	_start 							_stop 				_time 			_value	 _field 	 _measurement		tagKey
0 		2021-10-02T11:14:44.488Z 2021-10-02T	12:14:44.488Z 2021-10-02	T12:14:40.000Z 	1 		fieldKey	 measurementName 	tagValue


Then I changed the port in the sketch to the port of my webserver so I could log the request.

Code:
[getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 71 bytes
dumpio_in (data-HEAP): POST /api/v2/write?org=40c706b395812822&precision=s&bucket=boy HTTP/1.1
dumpio_in (data-HEAP): 2 bytes
dumpio_in (data-HEAP): \r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 21 bytes
dumpio_in (data-HEAP): Host: 192.168.1.145\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 111 bytes
dumpio_in (data-HEAP): Authorization: Token <<NOTSHOWINGTOKEN>>\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 26 bytes
dumpio_in (data-HEAP): Content-Type: text/plain\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 25 bytes
dumpio_in (data-HEAP): User-Agent: Arduino/1.0\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 20 bytes
dumpio_in (data-HEAP): Content-Length: 43\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 19 bytes
dumpio_in (data-HEAP): Connection: close\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 2 bytes
dumpio_in (data-HEAP): \r\n
dumpio_in [readbytes-blocking] 43 readbytes
dumpio_in (data-HEAP): 43 bytes
dumpio_in (data-HEAP): measurementName,tagKey=tagValue fieldKey=1i
dumpio_out
dumpio_out (data-HEAP): 180 bytes

Then I made a request with postman to the webserver so we can compare the requests.
Code:
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 73 bytes
dumpio_in (data-HEAP): POST /api/v2/write?org=40c706b395812822&precision=s&bucket=boy HTTP/1.1\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 111 bytes
dumpio_in (data-HEAP): Authorization: Token <<DIFFERENTTOKENSTILLNOTSHOWINGIT>>\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 26 bytes
dumpio_in (data-HEAP): Content-Type: text/plain\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 35 bytes
dumpio_in (data-HEAP): User-Agent: PostmanRuntime/7.28.4\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 13 bytes
dumpio_in (data-HEAP): Accept: */*\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 53 bytes
dumpio_in (data-HEAP): Postman-Token: <<NOPE>>\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 21 bytes
dumpio_in (data-HEAP): Host: 192.168.1.245\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 36 bytes
dumpio_in (data-HEAP): Accept-Encoding: gzip, deflate, br\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 24 bytes
dumpio_in (data-HEAP): Connection: keep-alive\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 20 bytes
dumpio_in (data-HEAP): Content-Length: 43\r\n
dumpio_in [getline-blocking] 0 readbytes
dumpio_in (data-HEAP): 2 bytes
dumpio_in (data-HEAP): \r\n
dumpio_in [readbytes-blocking] 43 readbytes
dumpio_in (data-HEAP): 43 bytes
dumpio_in (data-HEAP): measurementName,tagKey=tagValue fieldKey=1i
dumpio_out

Couple of things I found interresting; first, the line end in the influxdb POST from the teensy seems to be sent separate from the post line in comparison to Postman. Second, it seems that the body does not have \r\n, but since the body does get accepted by the database I'm not worried about this.

In order to be as complete as possible I have commented the empty println separating the headers from the body and re-posted to the database, this was the result:
Code:
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

This shows that the empty println is necessary in the request.

I have tried to remove https:// and add :443 to the Host when trying to post to AWS, which generated the response:

Code:
HTTP/1.1 301 Moved Permanently
Server: awselb/2.0
Date: Sat, 02 Oct 2021 12:42:08 GMT
Content-Type: text/html
Content-Length: 0
Connection: close
Location: https://www.influxdata.com/api/v2/write

Unfortunately I do not have more info about the hosted part of the server as it is being taken care of by Influx itself. I do have some links to the line protocol and API for you.

https://docs.influxdata.com/influxdb/v2.0/
https://docs.influxdata.com/influxdb/v2.0/api/#operation/PostWrite
https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/

@shawn regarding the examples remark, is this what you mean?


Code:
char server[] = "google.com";  // also change the Host line in httpRequest()

client.println("GET /index.html HTTP/1.1");
client.println("Host: www.pjrc.com");
client.println("User-Agent: arduino-ethernet");
client.println("Connection: close");
client.println();
client.close(); //If "Connection: close" make sure to close after printing and before stop to avoid harsh reset

I got this response:
Code:
connecting...
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Sat, 02 Oct 2021 13:13:13 GMT
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/index.html</code> was not found on this server.  <ins>That’s all we know.</ins>
 
What if you do not use client.println but client.print and put the \r\n at the end of the string:

client.print("POST /api/v2/write?org=ORG_HERE&bucket=BUCKET_HERE&precision=s HTTP/1.1\r\n");
 
What if you do not use client.println but client.print and put the \r\n at the end of the string:

client.print("POST /api/v2/write?org=ORG_HERE&bucket=BUCKET_HERE&precision=s HTTP/1.1\r\n");

Realistically speaking, even though it doesn’t arrive in the same packet that does not cause issues for TCP and he says it was accepted locally so there’s nothing wrong with the formatting of his message. I still maintain that this is purely because it’s being sent to the wrong IP Address due to the fact that he hasn’t included the subdomain in the server char array.
 
I just realized I have a spare ESP board laying around, so I decided to give Influxdb client for Arduino a try to exclude network issues on my end InfluxDB-Client-for-Arduino.
This library worked like a charm out of the box. Unfortunately I am not in the position to make use of wireless networking for this project.

A quick ping to eu-central-1-1.aws.cloud2.influxdata.com comes back with request timeout but shows me aed270f02d59711e9aaa902ee3ef9179-9aec589b0bda0555.elb.eu-central-1.amazonaws.com as an address. This combined with the server response header in my original post awselb/2.0 makes me believe I am hitting the Elastic Loadbalancer in AWS. What I don't understand is why the connection stops there and does not get forwarded.

@RoSchmi did you have any luck working with TLS rootCa verification after your git post? I can imagine this could be an issue as well since the ESP library uses BearSSL with certificates and my sketch does not.
 
Last edited:
@RoSchmi did you have any luck working with TLS rootCa verification after your git post? I can imagine this could be an issue as well since the ESP library uses BearSSL with certificates and my sketch does not.

I'm sorry but I don't exactly understand what you mean, please describe a little bit more to what you are exactly referring.
On my Github account 'RoSchmi' I have some projects for different platforms (e.g. Teensy, Esp32, Wio Terminal) storing sensor data to Azure Storage Tables. For the Teensy e.g. -https://github.com/RoSchmi/AzureDataSender_Teensy.
Do you mean this? The Teensy example works with bear_ssl and the http client from -https://github.com/khoih-prog/EthernetWebServer_SSL
 
Sorry, that was a very vague description on my end. I am referring to this issue on vjmuzik's NativeEthernet library git and was wondering if you got rootCa verification working with the NativeEthernet library. Regarding the EthernetWebServer_SSL library, I have tried to use my sketch with this library as well but got the same results as described in my initial post.
 
Make sure you update FNET to the latest version, there was a problem with DNS not resolving aliases and it sounds like that may be the issue now.
 
Both NativeEthernet and FNET are the latest version. I just did a test with a teensy 3.2 in combination with the EthernetWebServer_SSL and it works as expected when USE_ETHERNET is set to true in defines, so it uses Ethernet.h. Could it be that there's still some issues with DNS resolving in NativeEthernet?

I also checked if can post to influxdb with EthernetWebServer_SSL and Ethernet3 on my teensy 3.2, this functions properly as well.
 
Status
Not open for further replies.
Back
Top