C3 Protocol Generator

[Camel]

Hello,

I'm having a tinker with an old project and thought I'd share it. It deals with the problem of getting information from one device to another. This is such a common issue, nearly every non-trivial robotics project has to deal with it. Quite often the first impulse of a developer is to start writing code to serialize and deserialize structs so that they can be pushed out over a UART, or maybe start hacking at an ASCII based protocol (yuck). You then quickly realize it's very repetitive, tedious and error prone.

C3 contains a code generator (i.e. a dodgy python script) that reads a protocol description file to produce a C++ API that does all the dirty work. All you need to do is create the protocol description file, which is in JSON format, to describe what you need to send back & forth.
I've also thrown in a Javascript module that uses the protocol description file to serialise/deserialise your data inside NodeJS. You can use the serialport library to read/write to your Teensy.

The protocol description file lets you define enums, so you can send enum types back and forth. This is pretty useful for when you need to send back a status or a state of something where the number of states is finite . . . like the status of a finite state machine. It also has a settings singleton class, which of course, can be ignored if you don't want it. But, basically, in a system you will often want to tweak or make adjustments or calibrations, or whatever. These adjustments (aka settings) are global within software and, often, in my experience, are sent from a PC to the device via a UART and stored in FLASH memory. If you flag a message as a 'setting' in the protocol description, C3 will deserialise it directly to the appropriate fields of the settings class where it is globally accessible. I've found this very nifty, but don't expect it to be everyone's cup of tea. I wonder if it's perhaps out of scope for a protocol API . .hmm . . .

C3 will 'ack' messages if they have 'ack_required': 'true' in the protocol description. This means messages will not be removed from the pool of messages to be sent, until an acknowledgement is received for that particular message. This effectively guarantees the other end has received the message (or you get a timeout).

Some of the basic features of C3 are checksums (fletcher-16), zero calls to new/malloc (uses a memory pool), completely header-only library.

It compiles for Arduino UNO, Arduino Mega 2650, Teensy, other ARM chips and of course Linux/Windows/Mac/whatever. The amount of memory it uses is adjustable by changing the max packet length and size of the memory pool in c3.h (lines 35, 36). Normally I work with ARM chips with plenty of RAM and so I typically have max packet length to 256 and pool size to 2048. Never had any issues with my use case and that configuration. It should go well on a Teensy too. I've never really used C3 on an UNO except limited tests, but to even get it to compile and not eat all RAM I dialed it right down to 64 and 256 pool size. The memory pool holds packets before they are serialized, so if you hold off calling 'serialise', you may require a large pool so that called to 'create_packet' don't return nullptr. If you call serialise immediately after every 'create_packet' then you can make the pool size small (like packet size x2).

Some TODO items are

• improve serialisation performance (it serialises to a buffer then iterates over the buffer atm)
• do some actual error checking during code generation
• make the nodeJS code better and do examples

It has one dependency (etk). To install etk, just download it off github, create a folder in your Arduino library directory called 'etk' and copy over all the files from EmbeddedToolKit/inc/etk.

C3 Github

Anyway, reason I'm posting this is because of the off chance someone will find it interesting, or otherwise bother to dig through the code. Feedback & contributions welcome.

Cheers

 

[brtaylor]

Sounds a lot like protobufs, but presumably, lighter weight. Neat!

 

[Camel]

Yeah, kind of like protobufs, except it's designed with embedded in mind. Here's some example Arduino code based on the blinky.json file in the github repo

#include <c3.h>

typedef ccc::C3Link<class Coms, ccc::REMOTE> C3;

class Coms : public C3
{
	public:

	private:
		friend C3;

		void set_color_pack_handler(ccc::set_color_pack& pack) {
			auto color = pack.color;
			// do something to set the color
		}

		void set_pattern_pack_handler(ccc::set_pattern_pack& pack) {
			auto pattern = pack.pattern;
			// do something with pattern
		}

		void on_setting_changed(uint16 msg_id) {
			auto& settings = ccc::Settings::get();
			// save settings to flash/eeprom
		}
		void put(char c) {
			Serial.write(c);
		}

		uint8 get() {
			return Serial.read();
		}

		bool available() {
			return (Serial.available() != 0);
		}

};

Coms coms;

void setup() {
	Serial.begin(57600);
}

void loop() {
	auto& settings = ccc::Settings::get();
	// do something with settings.delay
	coms.read();
	coms.serialise();
}

It uses curiously reoccuring template pattern, so your communications class will be required to implement a few functions (put, get, available). These functions are used to bind the coms class to an interface. When you call coms.read(), it reads all available bytes. If a legit packet is read, it will be deserialised into a struct and the handler function in the coms class is called. When coms.serialise() is called, it serialises any packets in its pool and writes them using the put function.

To send a message, lets say you've got a message called 'hello' with a string field called text.

auto hello_msg = create_packet<ccc::hello_pack>();
if(hello_msg != nullptr) {
    hello_msg->text = "Hello world!";
}

This hello world message is now queued up ready to send. The next call to coms.serialise() will send it.

 

[RoboRepublic]

Why not use nanopb?

 

[Camel]

Why not use nanopb?

I hadn't heard of nanopb!

Just reading the nanopb docs on github, it seems like it's basically the same thing. Some points of difference are C3 is C++ not C, which is just a personal preference thing really and C3 also has a NodeJS library, I'm not sure if nanopb has an equivalent.
But clearly nanopb has a lot more development effort behind it.

 

[brtaylor]

The cool thing is that nanopb is protobuffers. So I'll use nanopb on Teensy 3.6, regular protobuf on a BeagleBone Black or a desktop (which can transpile to C++ code, Node.js, Rust, etc). It all works together. You can also setup your build system so the .proto file becomes a target and is rebuilt if it changes into whatever language source code you need (C, C++, JS, etc).

There are other cool benefits, like protobuf transpiled to C++ enables a lot of self reflection and the proto3 syntax gives tools to automatically convert protobuf binary messages to and from Json messages or files.

I definitely like having more options, which is why I'm excited to try your library when I get a chance. Nanopb would be worth checking out and seeing how it works, it might give you some ideas as well.

FWIW, I've also tried flatbuffers, but I found the decode / encode performance to not be better than protobuf and they took up much more space on the wire (protobuf does some neat things to compress message sizes).