Remote reboot / user code update

Status
Not open for further replies.

takedown

Member
Hello,

We will have 2 Teensy's in our installation controlling 1500-3000 leds (density to be determined). The 2 Teensy's will be running different user code and will run independently of each other. The installation will be supported remotely, the reset button unreachable and code will be updated from time to time to change the experience. We need to have a reliable way to run HalfKay and load new user code.

I see on this page:

https://www.pjrc.com/teensy/loader_cli.html

It mentions using a second Teensy to reboot the 1st Teensy. So our setup would look like:

Control Server 1
------------------
Server 1 USB to LED Teensy 1
Server 1 USB to Reboot Teensy 1
LED Teensy 1 connected from Reset to Reboot Teensy 1 Pin C7

Control Server 2
------------------
Server 2 USB to LED Teensy 2
Server 2 USB to Reboot Teensy 2
LED Teensy 2 connected from Reset to Reboot Teensy 2 Pin C7

Do you see any issues with this setup?

It appears that we cannot have all 4 Teensy controlled from the same server if we use the command line, is that correct?

Is the CLI compatible with 3.1? The CLI page did not list the option for the mk20dx256

Thank you for your help.
-Ron
 
The CLI is not (currently) compatible with the 3.1. There's a makefile hiding somewhere in the hardware folder inside your Arduino app (if you're on a mac) that shows a bit on how to use the GUI loader from the command line.

I'm not entirely sure you need to use another Teensy to reboot your main teensies. What type of USB HID style are you using for your LED-driver-teensys? I've been making a MIDI device, and for my purposes I've made a firmware loader that sends a very specific SysEx message to the Teensy to drop it into bootloader mode (using _reboot_TeensyDuino_() I think). It's easy enough to make my software send that MIDI message - essentially my firmware loader sets the GUI loader up the way it needs to be, tells it to load a particular file, then fires off my 'reboot' MIDI message - then code loading happens. All USB, no need for an extra Teensy.

If you can't call your LED Teensy's MIDI devices (via the menu), I'm sure there's a way to make it listen for a specific Serial message, or whatever else you might be able to use.
 
Hi MuShoo,

So if I understand correctly I would send a command over serial or ethernet to my teensy, it would listen for that special command and call the routine that places the Teensy into halfkay mode. Then I can pass args to the GUI loader to load a file and send it to the Teensy? I'll keep searching for that example in the Maefile.

I should have specified that I'm working on Windows, but the target control servers will be Linux.

I just noticed this:

https://github.com/PaulStoffregen/teensy_loader_cli/blob/master/teensy_loader_cli.c

Looks like Paul added support for 3.1 in the CLI, that's pretty cool. I'll have to give it a try.

Thanks.
 
Ah yes, guess I misspoke. The 3.0 (and apparently the 3.1, now) are supported from CLI but ONLY under Linux. So you might be OK in that regard!
 
There are 2 fundamental ways to approach remote rebooting.

The simple & cheap, but not 100% reliable approach involves sending a request to the Teensy. This is how Teensyduino implements the Upload button in Teensyduino. In Arduino's hardware/tools folder, you can find the "teensy_reboot" program which sends this message.

However, if Teensy has crashed with interrupts disabled or turned off its USB port or entered a low power mode where USB doesn't work, then it can't hear the auto-reboot request. Every Teensy has a pushbutton for exactly this reason, so you can recover from loading such programs.

The more expensive approach involves using another Teensy (or any remotely controllable hardware) to pulse the PROGRAM pin low, causing a reboot into programming mode. The command line loader features a "rebootor" example on a Teensy 2.0 which does this.... it simply listens for a request to reboot and pulses a pin low. Simple, except it adds more hardware. But if you want highly reliable rebooting, this is the only way that protects you from an unrecoverable situation where a remotely loaded program crashes in such a manner than it can't hear reboot requests.

Like all system design decisions, it's a matter of trade-offs. Perhaps you will only load well tested programs? Maybe the cost of accessing the pushbutton is high, but the odds of needing it are so low that saving the cost of extra hardware is worth the risk? Of course, for a system where you absolutely can't ever access the pushbutton, the extra cost is probably justified.

Both the "teensy_reboot" program in Teensyduino and the rebootor example with the command line loader are designed around rebooting a single Teensy.

The hardware rebootor example is probably the simplest way to expand to 4. Just edit the code and change one of the bytes in the message to indicate which of the 4 to reboot (and dedicate 3 more pins, obviously, to reboot the other 3 boards when the other messages come in). Then edit the command line code to send one of those 4 bytes, perhaps with a command line option to specify which one, or just compile 4 copies with the 4 different bytes. When you run it, the customized rebootor will reboot only 1 of the 4 Teensys, which will them get programmed with the code.
 
Ah yes, guess I misspoke. The 3.0 (and apparently the 3.1, now) are supported from CLI but ONLY under Linux. So you might be OK in that regard!

Well the commit happened 3 days ago, easy to miss ;)

Why is it supported only under Linux?
 
Hi Paul, thanks for your response.

We are not concerned with the added cost of another Teensy. We are just looking for the best way to handle remote updates and it sounds like additional hardware is the way to do it. I've ordered some extra Teensy's today to test with next week. We are considering putting the Teensy power supplies on a PDU port so they can be power cycled. We already have most of our equipment PDU controlled, so it would not be an issue to add additional hardware to the PDU.

My immediate questions are can we use the CLI with 3.1 (with the latest patch) on Windows to issue the reboot command to the Reboot Teensy and load an updated program to the LED Teensy? Or do I need to test with Linux to use the updated CLI code?

Thanks again.
 
I was able to build the latest teensy_loader_cli source on Ubuntu from github and to then program the teensy 3.1 from the command line. On to setting up a rebooter teensy for the final test. Thanks for the help.
 
The simple & cheap, but not 100% reliable approach involves sending a request to the Teensy. This is how Teensyduino implements the Upload button in Teensyduino. In Arduino's hardware/tools folder, you can find the "teensy_reboot" program which sends this message.

Paul: is there a way to do this without the Teensy loader running? I'm trying to setup remote updates on a headless Raspberry Pi. A second teensy is overkill for my application -- just trying to save myself the trouble of getting up and reaching into the cabinet when i want to try a new firmware.

Took a stab in the dark and tried pulsing DTR high briefly on ther virtual serial port (the same way you can reset Arduinos) but that doesn't seem to be how it's implemented for Teensy. Happy to submit a pull request for the teensy_loader_cli to support this if you can give me some direction on how it works.
 
Took a stab in the dark and tried pulsing DTR high briefly on ther virtual serial port (the same way you can reset Arduinos) but that doesn't seem to be how it's implemented for Teensy.
Teensy 3 doesn't use the DTR method. I suppose one could make DTR from a computer's serial or USB/serial port pulse the Teensy 3's reset line. Probably a little breadboard with a transistor and some resistors, or an opto-coupler 4 pin IC. But when the computer gets rebooted it likely will toggle the RTS and reset Teensy when you don't want it to. So you may need some combinations of serial port DTR/RTS. Or make a little board with an AVR tiny micro using its serial port and its GPIO line.

Another idea is to control the AC power to the Teensy's wall transformer - X10 module or one of the Ethernet based power on/off switch - connected to the LAN.
Lots of other ways to do this.

What many of us want is a means to remotely reprogram the Teensy via wireless 900MHz or WiFI. I have 900MHz working for remote control and programming AVR m328P boards. The issue seems to be how to open this enough to permit this to work but not enable rampant counterfeiting of Teensy 3.
 
Line of thinking was that since the serial port is emulated over USB, the serial port emulation on the Teensy could do whatever it wants with the DTR signal. There doesn't need to be a physical DTR pin connected to anything.

As Paul indicated, there's some sort of signal that you can send to the Teensy to tell it to reboot. Just got to figure out what it is.
 
Stevech: forgive me if I'm not being clear. I'm not interested in any kind of electrical reset mechanism. I'm interested to know what sort of message the teensy_reboot program sends to the Teensy to reset it. I'd love to pull that same functionality into the teensy_loader_cli.
 
Ah, I thought you wanted to reset a remote Teensy that has a nearby PC or LAN and you'd prefer to not have yet another Teensy just to reset a Teensy that's hung.

I think the mini54's loader program pulses the K20's reset pin. Via mini54's pin PB3.2 (see schematic http://www.pjrc.com/teensy/schematic.html).
My suggestion was to condition the PC's DTR or RTS from RS232 to 3.3V and pulse the reset pin from the PC. Or use a LAN-connected remote controlled relay/switch to do the same.
 
Broken Pipe

Here's what I've got so far. usb_control_msg fails with "error sending control message: Broken pipe". Any pointers?

Code:
int soft_reboot(void)
{
	usb_dev_handle *serial_handle = NULL;
	serial_handle = open_usb_device(0x16C0, 0x0483);
	if (!serial_handle) {
		char *error = usb_strerror();
		printf("Error opening USB device: %s\n", error);
		return 0;
	}

	char reboot_command[4] = {0xA9, 0x45, 0xC2, 0x6B};
	int response = usb_control_msg(serial_handle, 0x21, 9, 0x0200, 0, reboot_command, 4, 10000);

	usb_release_interface(serial_handle, 0);
	usb_close(serial_handle);

	if (response < 0) {
		char *error = usb_strerror();
		printf("Unable to soft reboot with USB error: %s\n", error);
		return 0;
	}

	return 1;
}
 
Code:
	char reboot_command[4] = {0xA9, 0x45, 0xC2, 0x6B};
	int response = usb_control_msg(serial_handle, 0x21, 9, 0x0200, 0, reboot_command, 4, 10000);
Changing that to
Code:
    char reboot_command = 134;
    int response = usb_control_msg(serial_handle, 0x21, 0x20, 0, 0, &reboot_command, 1, 10000);
Seems to work! I'll clean up my code and submit a pull request.
 
Hi Paul,

I finally returned to setting up the Teensy rebootor and I'm running into an issue with it actually rebooting the Teensy 3.1.

I connected the reset pin of the 3.1 to C7 of the 2.0 and hooked them both up to the PC through a USB hub. When both of them were connected to the hub I noticed that the blinky on the 3.1 would stop. If I only connected the 3.1 then blinky would run. I then updated the 2.0 with the rebootor code and when both were connected the 3.1 now ran it's blinky program as it should.

Does it make sense that prior to updating with the rebootor code that the 3.1 could not run it's program properly when connected to C7? Is it possible I damaged the 3.1 here and it's reset pad no longer works?

When I updated the 2.0 it went from running blink to having a solid led light lit.

I verified that both the rebootor and 3.1 showed up as unique devices using lsusb. This is on Ubunutu 14.04 with a patched cli to support the 3.1. I ran:

Code:
./teensy_loader_cli --mmcu=mk20dx256 -v -r ./blink_default.hex

and received back:

Code:
Teensy Loader, Command Line, Version 2.0
Read "./blink_default.hex": 10064 bytes, 4.0% usage
Hard Reboot performed
Waiting for Teensy Device...
 (hint: press the reset button)
Found HalfKay Bootloader
Read "./blink_default.hex": 10064 bytes, 4.0% usage
Programming.........
Booting

I need to push the reset button on the 3.1 to get it to past the "hint: press the reset button".

I'm not quite sure how to debug things past this now. For the time being I'll try another 3.1 and see if that works correctly.

Thank you.
 
I've been debugging things this morning and what I have found is that the rebootor code is working. I added some debug code to turn the LED off when the reboot message is received. I've also experimented with keeping C7 low after the reboot message is received and the 3.1 teensy stops running it's blinking program and the device is no longer visible in the usb device list. What I cannot figure out is how to get the 3.1 to boot into halfkay.
 
What I cannot figure out is how to get the 3.1 to boot into halfkay.

Just short the Program pin to ground, and then release it. As you can see on the schematic, that's all the pushbutton does. You can test this on a Teensy 3.1 without anything else, other than a piece of wire or paperclip.
 
Just short the Program pin to ground, and then release it. As you can see on the schematic, that's all the pushbutton does. You can test this on a Teensy 3.1 without anything else, other than a piece of wire or paperclip.

Hi Paul,

Maybe I'm misunderstanding how the reset is supposed to work. I have c7 from the 2.0 connected to the reset pad of the 3.1. I run the cli with -r and the rebootor code runs. However the 3.1 just continues to run its program and does not restart into halfkay, as a result the cli just sits and waits for the reset button to be pressed on the 3.1.

Is there additional debugging I can do, maybe on the 3.1 side?

Thanks for your help.
 
Pulling the Reset pad low causing the MK20 to reboot to your program.

Pulling the Program pin low causes the Mini54 to take control of the MK20 and reboot it into bootloader mode.

If your intention is to force the Teensy 3.1 to go into programming mode, connect to only the Program pin, not the Reset pin.
 
Thanks so much for the help Paul, that worked as designed. I now have a working setup where a 2.0 can set the 3.1 into bootloader mode and the teensy_loader_cli can remotely update the 3.1. I can move forward with this setup now that the remote support issue is handled.

It looks like I missed the note on this page: https://www.pjrc.com/teensy/teensy31.html about the 2 different reset signals.

Do you think it would be helpful to update this page:

http://www.pjrc.com/teensy/loader_cli.html

and mention on the 3.0/3.1 to use the PROG pin instead of the RESET pad when using the -r switch of the CLI?

I'm planning to run the 3.1 with external power, do I need to do the same for the 2.0 so that they share the same ground?

Thanks again.
 
Today I configured two 3.1's connected to the same 2.0 rebootor, modified the rebootor code and CLI to support specifying which to reboot and the tests went well. This is great, will have more to post about the project in the near future. We are doing a production test run next week.
 
Status
Not open for further replies.
Back
Top