Teensy 4.0 SPI.transfer causing program to hang/die when connected to SPI peripherals

Status
Not open for further replies.

cermak

Member
Hi!
I'm using a Teensy 4.0, and I'm running into an issue where my code gets stuck when I try to communicate over SPI. This likely is due to some kind of hardware mistake on my end (see below); however, I can't figure out what I'm doing wrong and would love any suggestions anyone might have for debugging this.

Here's a minimal example:

Code:
#include <SPI.h>
#define DAC_CS 10

void setup() {
  Serial.begin(115200);
  SPI.begin();
  delay(3000);
  Serial.println("Completed setup()");
}

void loop() {
  delay(3000);
  SPI.beginTransaction(SPISettings(1e6, MSBFIRST, SPI_MODE0));
  digitalWrite(DAC_CS, LOW);
  uint8_t send_buf[] = {0b10101010, 0b10101010, 0b10101010};
  uint8_t recv_buf[] = {0,0,0};
  Serial.println("Transferring");
  SPI.transfer( send_buf, recv_buf, 3);
  Serial.println("Transferred");
  digitalWrite(DAC_CS, HIGH);
  SPI.endTransaction();

}

When I run this on my Teensy 4.0 with nothing connected, I get the expected output:
"Completed setup()
Transferring
Transferred
Transferring
Transferred"​
ad nauseum every 3 seconds.

As soon as I connect the Teensy to my PCB, I get through the first two prints, but never reach the third printed line - the code seems to hang. My output is just:
"Completed setup()
Transferring"​

As the code seems to be hanging at the SPI.transfer call, I started throwing Serial.print statements into SPI.transfer(), lines 1549-1573 in SPI.cpp:

Code:
	while (count > 0) {
		Serial.print("first loop: count="); Serial.println(count);
		Serial.print("first loop: count_read="); Serial.println(count_read);
		// Push out the next byte; 
		port().TDR = p_write? *p_write++ : _transferWriteFill;
		count--; // how many bytes left to output.
		// Make sure queue is not full before pushing next byte out
		do {
			if ((port().RSR & LPSPI_RSR_RXEMPTY) == 0)  {
				uint8_t b = port().RDR;  // Read any pending RX bytes in
				if (p_read) *p_read++ = b; 
				count_read--;
			}
		} while ((port().SR & LPSPI_SR_TDF) == 0) ;

	}

	// now lets wait for all of the read bytes to be returned...
	while (count_read) {
		Serial.print("second loop: count_read="); Serial.println(count_read);
		if ((port().RSR & LPSPI_RSR_RXEMPTY) == 0)  {
			uint8_t b = port().RDR;  // Read any pending RX bytes in
			if (p_read) *p_read++ = b; 
			count_read--;
		}
	}

Now, with the Teensy plugged into my PCB the output is:
Completed setup()
Transferring
first loop: count=3
first loop: count_read=3
first loop: count=2
first loop: count_read=3
first loop: count=1
first loop: count_read=1
second loop: count_read=4294967295
second loop: count_read=4294967295
second loop: count_read=4294967295
second loop: count_read=4294967295
second loop: count_read=4294967295
second loop: count_read=4294967295
second loop: count_read=4294967295
second loop: count_read=4294967294
second loop: count_read=4294967293
second loop: count_read=4294967293
second loop: count_read=4294967293
second loop: count_read=4294967293
second loop: count_read=4294967293
second loop: count_read=4294967293
second loop: count_read=4294967293
second loop: count_read=4294967292
second loop: count_read=4294967291
second loop: count_read=4294967291
second loop: count_read=4294967291
...​

Somehow count_read was decremented past 0. I haven't quite reached the point of digging in to find out how SPI.transfer really works (like, what are port().RSR, port().SR, LPSPI_RSR_RXEMPTY, LPSPI_SR_TDF?), but I was hoping someone might have some idea based on this how I might have messed up.

Since this only happens when my Teensy is plugged into my PCB, I assume this is because I have misconfigured something on the SPI bus or with my SPI devices, but I'm at a loss as to what.


For reference, when I run the above minimal example on my Teensy 4.0 without it plugged into my board, I get:
Completed setup()
Transferring
first loop: count=3
first loop: count_read=3
first loop: count=2
first loop: count_read=3
first loop: count=1
first loop: count_read=2
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
second loop: count_read=1
Transferred
...​
 
On Teensy 4.0, MOSI is pin 11, MISO is pin 12 I believe, and that is what my design uses. If they were backwards though, how might this explain that result? My MOSI line is driving many devices, the MISO line is driven by only one device (an ADC), and is probably high-Z (but might be high or low - its in its startup state; i need to check the datasheet).
 
So far as I can tell this is not the case (or at least, MISO and MOSI are not crossed). Any advice on where I should look to learn about what this overflow practically means about the inputs? It would help me to understand how a badly-behaved MISO or MOSI line could cause this overflow. Thanks!
 
I have found that this issue continues even when only SCLK (pin 13) and GND are connected to my custom PCB. This occurs when my custom board is both powered or unpowered. The clock line is driving 36 SPI-compatible ICs (34 ADG1414s, 1 LTC2668, 1 MAX11156).

Weirder still, if instead of placing my Teensy into the socket on my PCB, I connect the Teensy to the socket via 6" long jumpers, I do not have this issue with SPI.transfer failing, and my oscilloscope shows correct SPI operation.

Is there any place I should be looking for physical constraints on the SPI lines? I think the thing I'm unclear on is how exactly the SPI interface differs from bitbanging out an SPI command via digitalWrite and digitalRead. So far as I can tell, these functions do not fail when I run them, even with the Teensy 4.0 connected to my board. Why is SPI.transfer() different? At worst, I would expect corrupted information off the MISO line.

Thanks again!
 
I'm not an SPI expert here but you never said what device you have hooked into the T4 SPI lines. You mention you plug the T4 into your PCB but we don't know what your PCB or what you have hooked up, like are there any other devices hooked up, did it work at all with any other Teensies like the T3.6 etc.... We see the code but have no clue what you hooked into it. Some SPI dac's require commands to be sent first the data transfer occurs. Did you read the data sheet to see if that is the correct way to do it.

Just guessing here.
 
From the looks of it, from your debug output, somehow it received more bytes than it should? That is
Code:
void SPIClass::transfer(const void * buf, void * retbuf, size_t count)
{

	if (count == 0) return;
    uint8_t *p_write = (uint8_t*)buf;
    uint8_t *p_read = (uint8_t*)retbuf;
    size_t count_read = count;

	// Pass 1 keep it simple and don't try packing 8 bits into 16 yet..
	// Lets clear the reader queue
	port().CR = LPSPI_CR_RRF | LPSPI_CR_MEN;	// clear the queue and make sure still enabled. 

	while (count > 0) {
		// Push out the next byte; 
		port().TDR = p_write? *p_write++ : _transferWriteFill;
		count--; // how many bytes left to output.
		// Make sure queue is not full before pushing next byte out
		do {
			if ((port().RSR & LPSPI_RSR_RXEMPTY) == 0)  {
				uint8_t b = port().RDR;  // Read any pending RX bytes in
				if (p_read) *p_read++ = b; 
				count_read--;
			}
		} while ((port().SR & LPSPI_SR_TDF) == 0) ;

	}

	// now lets wait for all of the read bytes to be returned...
	while (count_read) {
		if ((port().RSR & LPSPI_RSR_RXEMPTY) == 0)  {
			uint8_t b = port().RDR;  // Read any pending RX bytes in
			if (p_read) *p_read++ = b; 
			count_read--;
		}
	}
}
You were printing out that count_read < 0 ...

Which can only happen up at: count_read-- in the first loop.
Which should not happen as the first thing we do is to clear the Receive buffer (RRF), unless there is something else going on...

You might try a quick hack. Change in the first loop:
Code:
			if (count_read && ((port().RSR & LPSPI_RSR_RXEMPTY) == 0))  {
				uint8_t b = port().RDR;  // Read any pending RX bytes in
				if (p_read) *p_read++ = b; 
				count_read--;
			}
And see if that stops your hang
 
Hi mjs513. I unfortunately cannot post my full circuit design. I also have not tried this with other Teensies (though have used Teensies 3.2,3.5 and 3.6 extensively in previous projects), but I also need to use the Teensy 4.0 for this project as I need the speed. As I mentioned in a previous post, the Teensy SPI lines are hooked up to a variety of ICs on my board, specifically:
SCLK (pin 13) -> 34 ADG1414 ICs, 1 LTC2668 and 1 MAX11156
MOSI (pin 11) -> 2 ADG1414s, 1 LTC2668, and 1 MAX11156
MISO (pin 12) <- 1 MAX11156

Importantly, my issue does not appear to be with malfunctioning ICs, but rather code that is not running as expected on the Teensy when the clock line and GND alone are connected.

I have pored over the datasheets for these chips to try to understand why this fails. SCLK is in all cases connected to clock inputs on these chips; MOSI is connected to these chips SDI pins; MISO is connected to the ADC's DOUT pin.

(Datasheets: https://www.analog.com/media/en/technical-documentation/data-sheets/ADG1414.pdf, https://www.analog.com/media/en/technical-documentation/data-sheets/2668fa.pdf, https://datasheets.maximintegrated.com/en/ds/MAX11156.pdf)
 
Don't have a lot of experience with these devices but just a couple things to look at.
1. Assuming you have different CS pins for the DAC and the ADC.
2. In the case of the DAC suggest that you read page 16 operation for the DAC. Basically it showing:
The 4-bit command, C3-C0, is loaded first, followed by the 4-bit DAC address, A3-A0, and finally the 16-bit data word in straight binary format. For the LTC2668-16, the data word comprises the 16-bit input code, ordered MSB-to-LSB. For the LTC2668-12, the data word comprises the 12-bit input code, ordered MSB-to-LSB, followed by four don’t-care bits. Data can only be transferred to the LTC2668 when the CS/LD signal is low. The rising edge of CS/LD ends the data transfer and causes the device to carry out the action specified in the 24-bit input word. The complete sequence is shown in Figure 3a.
There is a lot more to it than what I quoted for the chip.

Also suggest that you play with the chips individually before trying to connecting them all together
 
Thanks for your help!

mjs513 -
I have read this datasheet a good several dozen times. I dont understand how this relates, as my problem is not writing to the DAC, it is with the SPI output of the Teensy itself. My problem occurs in the Teensy 4.0's SPIClass::transfer(const void * buf, void * retbuf, size_t count) function when I have only connected SCLK and GND pins to my board.

Your suggestion to play with the chips individually is a good one that I will explore if I can; however, these are DFN chips and not easy to hand-solder, hence I'm outsourcing my assembly and don't have as much flexibility as I'd like.

KurtE - I tried your modification to SPI.cpp. When the Teensy is connected to my PCB, count_read seems to reliably go from 3 to 0 in the first cycle. I'm not totally clear how or why this ought to happen, since my impression is that we can't possibly have received three bytes in the duration we only pushed 1 out, since they're on the same clock...

17:07:39.531 -> Transferring
17:07:39.531 -> first loop: count=3
17:07:39.531 -> first loop: count_read=3
17:07:39.531 -> first loop: count=2
17:07:39.531 -> first loop: count_read=0
17:07:39.531 -> first loop: count=1
17:07:39.531 -> first loop: count_read=0
17:07:39.531 -> Transferred
17:07:39.560 -> Transferring
17:07:39.560 -> first loop: count=3
17:07:39.560 -> first loop: count_read=3
17:07:39.560 -> first loop: count=2
17:07:39.560 -> first loop: count_read=0
17:07:39.560 -> first loop: count=1
17:07:39.560 -> first loop: count_read=0
17:07:39.560 -> Transferred
17:07:39.593 -> Transferring
17:07:39.593 -> first loop: count=3
17:07:39.593 -> first loop: count_read=3
17:07:39.593 -> first loop: count=2
17:07:39.593 -> first loop: count_read=0
17:07:39.593 -> first loop: count=1
17:07:39.593 -> first loop: count_read=0
... and so on

(Note this is with exactly the same code as I posted previously, but added "pinMode(DAC_CS, OUTPUT);" to setup() and "delay(3000);" changed to "delay(30);" in loop(). )

However, while this fixes the infinite loop issue, the SPI lines are still not behaving appropriately. Here's CS, CLK, and MOSI (yellow, purple, blue) without my board attached, then with. Note that these aren't on the same timescale. Long story short, there is mysteriously extra data and the clock goes on too long (and idles high, not low). Note this same behavior occurs with only GND and pins 10 (DAC_CS) 11 (MOSI) and 13 (SCLK) connected.
SDS00004.jpg
SDS00006.jpg

What could my board be doing to mess up the Teensy's SPI output? These lines aren't driven when the teensy is not connected. Could it have to do with load on the clock or MOSI?
 

Attachments

  • SDS00005.jpg
    SDS00005.jpg
    86 KB · Views: 122
Last edited:
mjs513 -
I have read this datasheet a good several dozen times. I dont understand how this relates, as my problem is not writing to the DAC, it is with the SPI output of the Teensy itself. My problem occurs in the Teensy 4.0's SPIClass::transfer(const void * buf, void * retbuf, size_t count) function when I have only connected SCLK and GND pins to my board.
Ok, here is where I am getting confused then. In your first post you stated that with nothing connected to Teensy "When I run this on my Teensy 4.0 with nothing connected, I get the expected output:" but when you connect it your PCB "As soon as I connect the Teensy to my PCB, I get through the first two prints, but never reach the third printed line - the code seems to hang". Assumed by this statement you had all lines SCK, MISO, MOSI, CS connected to your PCB with the chips on them. That was why I made the statement about the interface to the DAC. Thought you were trying to get something from one of the chips you listed Only in you last post where you stated "Teensy 4.0's SPIClass::transfer(const void * buf, void * retbuf, size_t count) function when I have only connected SCLK and GND pins to my board. " is making what you are testing clearer.

As for your note, yeah you have to do that, missed that in your code. From the Teensy SPI page:
Code:
  // set the slaveSelectPin as an output:
  pinMode (slaveSelectPin, OUTPUT);

Sure @KurtE can explain the details for the clock and the transfers. I usually look at the CS if it low the transfer occurs and the data that is transmitted and recd. @KurtE convinced me to get a Logic Analyzer for things like this and its a lot easier to see whats going on.
 
Perhaps your PCB is causing very fast reflections on the SCK line that are causing glitches inside the T4 SPI controller. I think SPI.begin sets up the port pins for SCK and MISO and MOSI for the strongest drive (IOMUX_PAD_DSE(7)). You might try choosing one of the weaker drive strengths that are commented out in SPI.begin().

If you are connecting SCK to all 34 ADG1414s, that's a lot of capacitive loading on SCK (9pF x 34). If I had designed that board, I would have put a buffer between the SCK input and all those analog multiplexers.

Weirder still, if instead of placing my Teensy into the socket on my PCB, I connect the Teensy to the socket via 6" long jumpers, I do not have this issue with SPI.transfer failing, and my oscilloscope shows correct SPI operation.

In this case, the jumper wires may be changing the timing of echos from the PCB.
 
I just ran a quick LTSpice model of a 1MHz clock driving 300pF through a 20-ohm resistor (20 ohms is the nominal output impedance of the SCK pad at maximum drive strength). The model showed a current spike of about 150mA at the clock edges. The combination of low output impedance and high capacitive loading is certainly putting a strain on the pad driver and any circuitry on the same power supply traces inside the MCU.

I added an inductance of 6uH in series with the resistor---about what I measured for the inductance of a 6" stranded jumper wirre--and the peak currents went down to about 16mA. That might explain why things worked when you connected with jumper wires.

I think my earlier characterization of the problem as 'reflections' was poor choice of terms.

I think it's also important that the capacitive loading causes both positive and negative current spikes through the SCK pin. Depending on the on resistance of the MOSFETS driving the pin, those current spikes could be doing bad things inside the chip.
 
Sorry, some of the others here can (and have) explain the electrical stuff a lot better than I can.

There are lots of details here that I am unsure of on how they may influence your setup.

In particular the idea of having one SPI setup drive 36 SPI-compatible ICs (34 ADG1414s, 1 LTC2668, 1 MAX11156).

Again not sure of the details, like are the 34 ADG1414s setup each one connected to the 3 pins or are they daisy chained? I don't see any CS pin associated with them, SO not sure what is controlling signals going to them versus your other chips. Are they all 3.3v? If daisy chained do you have PU resistors to some of the output pins?

Also not sure how these are all powered? Do you have these powered external or is this some how powered by Teensy?

Again I am not much of a hardware guy so I mainly tinker. But again if it were me, I would try smaller subsets of this stuff to see if somehow they are interfering or maybe ???
 
I tested my hypothesis about the pad drive of a highly capacitive load by connecting two 150pF capacitors from SCK to ground.

I then ran the OP's original simplified test program. Here are the results:
with "uint32_t fastio = IOMUXC_PAD_DSE(7) | IOMUXC_PAD_SPEED(2);"--the default setup in SPI.begin. 300pf on SCK Program hangs after first 'Transferring'
with "uint32_t fastio = IOMUXC_PAD_DSE(7) | IOMUXC_PAD_SPEED(2);"--the default setup in SPI.begin. 150pf on SCK Program hangs after first 'Transferring'
with "uint32_t fastio = IOMUXC_PAD_DSE(2) | IOMUXC_PAD_SPEED(1);"--lower drive and speed in SPI.begin. 300pf on SCK Program runs properly and continuously

This seems to confirm my hypothesis that setting up the SPI pad drive properly will correct the issue of failing SPI transfers when SCK is connected to a large capacitive load.
 
Thanks so much everyone for all your help!

mjs513 - Sorry, that was definitely info that should have all been in the first post! I'll definitely work on explaining my problems better!


mborgerson - thanks so much! this definitely fixes my test program, and provides some insight to me as to how I might run into problems here. I'm still running into some issues in my actual program (haven't yet made a minimal program that reproduces the next outstanding bug), but I think this strongly suggests I need to fix the clock line by adding some kind of buffer. You mentioned that "If I had designed that board, I would have put a buffer between the SCK input and all those analog multiplexers." - do you have a go-to chip you'd recommend off the top of your head?

I don't fully understand a lot of the NXP pdf you referenced (though I have a feeling I'll be going over it a lot in the next couple days) - are the parameters you suggested the most extreme in terms of limiting drive strength and speed? I see there's also a slew rate parameter, but I'm not clear how to set that.

I have made circuits with SPI before, but never with anywhere near this many components (typically 1-4).


KurtE - I have daisy-chained the ADG1414s data lines (the Teensy's MOSI only drives the first one) with pullup resistors on each SDO->SDI connection, but they all receive the same clock line, and there is a "CS"-like input called "~SYNC" that I drive from the Teensy that pulls every ADG1414's ~SYNC pin low synchronously, during which I clock out 8*34 = 272 bits, then set ~SYNC high to set the switches.

All devices are powered off a DC/DC converter module (TMR3-0521) that is running off the USB input power. I'm also using the Teensy's 3.3V regulated line for providing logic-level references for a few chips, but it shouldnt be providing any significant amount of current (definitely order of magnitude below the 250mA limit). In my testing so far, changing the power connections has made 0 difference in terms of my SPI problems. For example, it doesnt matter if I connect or disconnect that 3.3V pin, or if I enable or disable the DC/DC converter.

You advice on trying smaller subsets is a good one and mirrors mjs513's earlier advice; I'm just not 100% sure how to do it with these DFN chips, since I can't just breadboard them, and even soldering them isnt something I'm really sure how to do properly - my soldering limit is basically a hand soldering iron with 0402s and TSSOPs).
 
My go-to choice for buffers for a decade or two has been one of the single-channel versions of the 74C125. I have used the NC7SZ125M5X in a number of designs.
However, I most often used it as a transition between 5V and 3.3V signals because of the 5V-tolerant inputs. There may be other buffers that will handle the large load capacitance better than this chip.
 
Status
Not open for further replies.
Back
Top