Smashing Atoms with a Teensy

Cowlander

Member
I'm on the operations staff of the Cyclotron Institute at Texas A&M University. We have two cyclotrons and a variety of ion sources, beam lines and auxiliary systems. A good deal of it is under computer control. The control system is my baby. It has a LabView front end control console and over 100 embedded controllers to handle power supplies, read vacuum, control valves, etc. etc etc. We currently use a Rabbit 3200 module for our embedded controller but we have no backup. I've been building and testing a Teensy 3.5 as a possible replacement. Our standard card has 6 serial ports, an SPI port to control relays and DACs, and a general digital I/O port ( 13 gpio bits ). The Teensy 3.5 is a good fit when combined with a Wiz850 network module. So far, there have been no serious issues while bench testing the prototype. I hope to place it in a high stress environment to test how it reacts to radiation, high energy sparking, and the heavy hand of scientists.
 
Pictures it is.
Teensy 3-5.jpg
 

Attachments

  • Box.jpg
    Box.jpg
    168.6 KB · Views: 291
Looks great! I wasn't familiar with the Rabbit RCM3200 so I looked it up. Don't tell Paul how much it costs, or he might raise the price of the Teensy :)
 
Awesome looking unit Cowlander! Hope your commissioning goes well.

Out of curiosity, do you use EPICS for a control system, or is it all Labview?

I did the installation of a synchrotron storage ring RF system a few years ago, you picture is bringing back memories...
 
So far, so good! It has been running for a couple of weeks with only minor teething problems.

The front end is just Labview, but anything that connects to the network can send commands and receive data
 
Funny I just came across this post -- we routinely use Teensy 3.2/3.6 (and soon 4) to operate devices being tested in the Texas A&M Cyclotron beamline (among several facilities we visit). They're convenient for us because they've got such a great combination of rapid development time, convenient footprint, and high performance. I'm happy to hear Cowlander is developing control routines for the facility using the same devices. Very cool!

Here's a Teensy 3.6 being used to operate a multi-Tb flash memory chip under irradiation at Texas A&M:
IMG_20180430_011447.jpg
(photo available for public posting)
 
Very nice photo DukeBlue! Were you able to make any significant errors appear in the chip?

You're reminded me of a quick experiment I did about a decade ago. At the time I was commissioning some large (from memory something like 100keV) industrial X-ray units, and we ended up having some free time while waiting for a colleague.
I decided being unsupervised like this was a once in a lifetime opportunity to find out if putting my electronics through airport scanners (of considerably less keV) was actually dangerous to my precious data.
So I found an old SD card, and loaded it up with all the large files I could find, whilst noting both the CRC & MD5 hashes for them, so that I could later detect if the X-rays flipped even a single bit.
Then I shut down the beam, and used a piece of paper to position the SD card carefully in the dead-centre of the collimated beam, before activating it again.

I ran it for 30 seconds, then shut down and compared the data, no errors.
I ran it for another 5 minutes and compared again, still nothing.
I was about to run it for even longer, when my impromptu experiment was noticed by the machine's designers. I was worried that they'd wrap me over the knuckles for messing with it, but to my delight they also got interested in my experiment.

They tapped in various arcane maintenance codes to the tube power supply, which overrode default tube current and voltage limits, as well as removing all attenuating filters. The lowly visitors (including myself) were made to stand far back*, and they opened the shutter and ran it for a few minutes at the herculean level they'd put it in.

We shut down again, and I compared the data once more. *No changes!*

That was maybe 2007, so flash technology has probably changed a lot since then. But I remember being much more at-ease walking though airport security with my laptop after that point.

Cheers,
Gavin


* I sincerely doubt there was any real danger, even backscatter off the SD card would have been fairly minimal, and we all had dosimeters and survey meters. But ALARA dictates being overcautious just in case.
 
Very nice photo DukeBlue! Were you able to make any significant errors appear in the chip?

Yes, especially with highly-scaled flash (planar flash from 2014 or 3D flash) it's pretty easy to do so. In fact they won't even operate error-free outside of the beam, so a heavy ion beam can easily turn a 0 into a 1. While we are characterizing all kinds of interesting effects in the error rates of 3D flash, we're primarily trying to make sure there aren't any destructive effects on a particular device that would preclude use in a space radiation environment. The bit errors are a lesser problem.

You're reminded me of a quick experiment I did about a decade ago. At the time I was commissioning some large (from memory something like 100keV) industrial X-ray units, and we ended up having some free time while waiting for a colleague.
I decided being unsupervised like this was a once in a lifetime opportunity to find out if putting my electronics through airport scanners (of considerably less keV) was actually dangerous to my precious data.
So I found an old SD card, and loaded it up with all the large files I could find, whilst noting both the CRC & MD5 hashes for them, so that I could later detect if the X-rays flipped even a single bit.
Then I shut down the beam, and used a piece of paper to position the SD card carefully in the dead-centre of the collimated beam, before activating it again.

I ran it for 30 seconds, then shut down and compared the data, no errors.
I ran it for another 5 minutes and compared again, still nothing.
I was about to run it for even longer, when my impromptu experiment was noticed by the machine's designers. I was worried that they'd wrap me over the knuckles for messing with it, but to my delight they also got interested in my experiment.

They tapped in various arcane maintenance codes to the tube power supply, which overrode default tube current and voltage limits, as well as removing all attenuating filters. The lowly visitors (including myself) were made to stand far back*, and they opened the shutter and ran it for a few minutes at the herculean level they'd put it in.

We shut down again, and I compared the data once more. *No changes!*

That was maybe 2007, so flash technology has probably changed a lot since then. But I remember being much more at-ease walking though airport security with my laptop after that point.

Cheers,
Gavin


* I sincerely doubt there was any real danger, even backscatter off the SD card would have been fairly minimal, and we all had dosimeters and survey meters. But ALARA dictates being overcautious just in case.

I can't recall offhand what that energy level will do since we rarely do xrays, but its entirely possible you created huge numbers of errors in the memory that were handled by the ECC of the memory controller. Raw NAND testing is much easier since you have low level control without any ECC. Testing of an SSD or SD card is a pain because you can't see behind the ECC and thus is works until suddenly it just doesn't (when you have corrupted things so badly the controller can't figure it out).
 
I can't recall offhand what that energy level will do since we rarely do xrays, but its entirely possible you created huge numbers of errors in the memory that were handled by the ECC of the memory controller. Raw NAND testing is much easier since you have low level control without any ECC. Testing of an SSD or SD card is a pain because you can't see behind the ECC and thus is works until suddenly it just doesn't (when you have corrupted things so badly the controller can't figure it out).

Aah, good point.
 
It's a small world. Dukeblue, do you ever see any single event upset or other radiation phenomena in the Teensy?

Nope. Really wouldn't expect to with your heavy ion beam and its typical energies and ranges, and we keep the Teensy out of the beam line anyway. Some stray neutrons could maybe upset the Teensy but Ive never seen it. When we do proton testing it would be far more of an issue, but I haven't yet done one with a Teensy.
 
It's a small world. Dukeblue, do you ever see any single event upset or other radiation phenomena in the Teensy?

Small update since this thread got bumped. The T4.0 is my current controller for certain digital components that need radiation testing. I've been killing them off occasionally when used near, but not in, the beam line (at LBNL, not TAMU, but similar energies). I'm guessing the bootloader is getting corrupted and the parts are cheap enough that I haven't bothered to look into it too much, but it was never an issue with T3.1/3.2/3.6 which worked flawlessly near the beam.
 
And I've just finished an upgrade using the T4.1 ( easier layout, native Ethernet ), currently being tested. I'll need to try it in one of our hot spots.


Small update since this thread got bumped. The T4.0 is my current controller for certain digital components that need radiation testing. I've been killing them off occasionally when used near, but not in, the beam line (at LBNL, not TAMU, but similar energies). I'm guessing the bootloader is getting corrupted and the parts are cheap enough that I haven't bothered to look into it too much, but it was never an issue with T3.1/3.2/3.6 which worked flawlessly near the beam.
 
And I've just finished an upgrade using the T4.1 ( easier layout, native Ethernet ), currently being tested. I'll need to try it in one of our hot spots.

I would be very curious in your results. I'll be at TAMU next week and might have a chance to run a Teensy in the K150 but TBD, it's low priority.
 
Update: The Teensy 3.5 based controllers have been retired, and the new cards based on the Teensy 4.1 are coming into service.
They never gave me any trouble in the 3.5 years they were active. The 4.1 controllers are slated to be used for all new power supply control.
 
Back
Top