Periodic click when saving ADC input to SD card

YasarYY

YasarYY

Active member
Hi. On Teensy 4.1, when I acquire the audio from the ADC input, and save it to SD card, a periodic click occurs.

I perform high sampling, and click period changes by sampling rate. At sampling rate 176400 Hz period is ~1.86 seconds, at 352800 Hz ~0.93 seconds.

That's because click occurs at a constant sample count period, 327680 16-bit samples, being 655360 bytes or exactly 640 kb. Click may occur rarely sooner.

This relates to SD card writing, specifically to flush operation, which happens when write buffer reaches 640 kb, or rarely sooner.

Flush operation draws considerable amount of additional current ~100 mA for ~20 ms according to below post:

This current draw pulls ADC's 3.3v a bit down for an instant, and since ADC reference is also that 3.3v line, ADC measures all-ones for that instant. Zoomed in view of record (at 352800 Hz) of that instant is below.

1766734966571.png


When I searched for this, I found several posts, but cannot find a well-defined solution.

I tried manually calling flush(), at different periods, but it just changes the period of click accordingly.

Below post says "capacitors cross the 3.3V supply to uSD may minimize impact", but how?

Can someone please explain what can be done to mitigate this issue? Or I'll have to patch this spikes at software. I tried to be brief, if you need more details let me know.

Thanks,
Yasar
 
SDcard pull heavy currents when writing and erasing, unless you have separate regulation for ADC and SDcard this will cause noise spikes on the ADC readings - adding lots of bulk capacitance can help (millifarads, not microfarads) by attenuating the voltage spikes.
 
Thank you for the response Mark.

To where that bulk capacitance should be added? Between any 3.3v and ground lines, or do you recommend a specific place? 🤔
 
Since those infrequent updates are too much for the supply, try decreasing the interval. More writes to the SD card but each one requires less energy. You might get lucky and find a rate which works.
 
For 640KB, are you buffering in PSRAM? Just a suggestion, but you might want to look at Sdfat example program TeensySDIOLogger. It shows what I think is a set of best practices to avoid blocking in SD write:
  • use Sdfat's RingBuf (you need < 50KB buffer for 700KB/sec data rate)
  • preAllocate file to avoid sector search/erase during logging
  • always write to SD in 512-byte (1 sector) chunks
  • always test for SD.isBusy()==false before writing
 
What I did with a T4.1 + custom ADC board is to have on ADC-board an own 5V to 3.3V LDO and using as 5V source the USB-host 5V output. This has a 100 uF Capacitor, and as a bonus, can be switched on/off easily via software. Only the digital part of the ADC uses the 3.3V from Teensy board.The disk-writing interference never disappeared complete, but was acceptable low level.
For a 3.3V ADC board you can interleave a small 5V to 3.5V LDO. Not to forget, if there is a preamplifier, you are better of to run Analog part from battery and not from teensy.
 
I'm interested in this (as in avoiding) as I am merging the adc_dma.ino and fast_logger.ino examples at 815kHz ADC.
If this is an issue then supplying the Teensy 4.0 separately (different USB port on laptop) from the Adafruit microSD breakout should fix it (?)
The USB 2.0 spec of 500mA doesn't leave a lot of headroom with a Teensy and a couple of hats.
mF seems like a lot for this...
I also use 3.7V litium flat packs for powering Teensys and Picos.
 
Thank you for the responses. I tried all of them, and below is the summary of the test journey.

TLDR; none of the suggestions worked, so I'll patch click spikes at the software.

* Place capacitors
I placed a capacitor to 3.3v line near pin 23, near pin 24 and near sd region, one at a time.
Flush starts taking 50+ ms time and creating click spikes after 1~2 minutes of recording, with a period of ~0.93 seconds (described at the first post).
Code: 
cap value                           flush delay [ms]              clicks at
none                                51~53                         flushes
1000/470/100 uF                     51~53                         flushes

* Separate ADC board
It is deemed overkill as a fix for periodic clicks

* Decreasing flush interval with manual flush() calls
Clicks have the same period with flush() calls, and are shorter. Previously seen 50+ ms delays with the same period still persist, I named them episodics.
Episodics occur at every few flush() calls. They are at the issue's original interval, at 352800 Hz ~0.93 seconds, 640 KB data periods, probably caused by the search for free clusters. SD card I use is exFAT and 128 GB, having cluster size of 128 KB (e.g. a 1 byte file takes 128 KB). So, 50+ ms flush delays at 640 KB data periods might mean that the library is allocating 5 clusters at a time when needed.
Code: 
flush() in how many update()s       flush delay [ms]              clicks at
32                                  2~7, episodics 52             flushes
16                                  1~5, episodics 52             flushes
8                                   1~4, episodics 52             flushes
4                                   1~3, episodics 53             flushes

* Use Sdfat's RingBuf
RingBuf using (Teensy's default?) File class which is in ...\cores\teensy4\FS.h
RingBuf.sync() is used instead of RingBuf.writeOut(512). Because at every audio update(), I write 2048 bytes to RingBuf, being exactly 4 sectors. So sync() already syncs an exact number of sectors.
Tried 100 * 512 and 500 * 512 bytes of buffer sizes (100 and 500 sectors), behaviors are the same.
Delays are much more stable. Episodics occur at every few sync() calls.
Code: 
sync() in how many update()s        sync delay [ms]               clicks at
32                                  2.8, episodics 55             syncs
16                                  1.4, episodics 54             episodics only
8                                   <1, episodics 53              episodics only
4                                   <1, episodics 53              episodics only

* Preallocate file to avoid sector search/erase during logging
File class doesn't have isBusy() and preAllocate().
RingBuf using (SdFat's default?) FsFile class which is in ...\libraries\SdFat\src\FsLib\FsFile.h
Program using SdFs class which is in ...\libraries\SdFat\src\SdFat.h (instead of SD class in SD.h)
Results have not changed. To be sure the file is preallocated, I removed truncate() call after file writing is completed, and the resulting file was at the preallocation size.

* Use FsFile and SdFs without RingBuf
Results have not changed.

* Always write to SD in 512-byte (1 sector) chunks
Already doing, no effect on click.

* Always test for SD.isBusy()==false before syncing
Results have not changed.

* Feeding Teensy 4.0 separately from the Adafruit microSD breakout
I'm using Teensy 4.1 which has an on-board Micro SD Socket.

* Patch click spikes at the software
Not a solution, but I spent enough time on searching for a decent solution. Patching 6-7 values of "one" by using nearby values is the workaround I'll use.
 
You say you are writing 2048 bytes at a time. That is a multiple of 512, but have you tried 512? It MUST be 512 to avoid the long SD blocks.
 
Maybe the click sound is due to ground current flowing between Teensy and whatever other hardware you have connected to Teensy?

It's easy to think of the 3.3V power fluctuating, and that can indeed be the problem. But since adding a capacitor didn't help, if you still want to explore the next step is to try eliminating ground loops. Unintentional ground current loops are often the cause of these sorts of unwanted noises. Many times on this forum we've heard of problems with undesired sounds that went away once the other ground connection disappeared.

If Teensy is running from USB power, and if your program can work without USB connected, the first thing to try (because it's so easy) is running Teensy from a 3.7V or 4.5V battery. Even if running from batteries is completely impractical for your final project, as a quick test it can really help to better understand the true cause of the problem.

Alternately, if the other gear that's creating the signal has a ground connection (other than the ground wire to Teensy), if you have a way to run it from a battery so it's ground isn't connected to anything else other than Teensy, that can give you another easy way to test.

If neither of these are feasible, you could try using a ground loop isolator for the audio (alternate Amazon product if SparkFun is out of stock), or a USB isolator for Teensy's USB connection. If you buy the Hifime product, make sure to get version 2 which has been verified to work with Teensy 4.0. Their older version 1 worked partially but got confused by quick disconnect and reconnect when uploading new code to Teensy.

However you test, if the click sound goes away when you have no other ground connection, then you can focus your effort on ground loop current rather than the 3.3V power.
 
Last edited:
joepasquariello said:
You say you are writing 2048 bytes at a time. That is a multiple of 512, but have you tried 512? It MUST be 512 to avoid the long SD blocks.
Click to expand...
Yes, I tried writing buffered data to file by several writeOut(512) calls, waiting while the file is busy each time.
All writeOuts was successful, below code never printed "writeOut not Ok", but behavior didn't changed.
C++: 
bool writeOutOk = true;
while (rb.bytesUsed() != 0) {
    while (frec->isBusy());
    if (rb.writeOut(512) != 512) writeOutOk = false;
}
if (!writeOutOk) Serial.println("writeOut not Ok");
 
PaulStoffregen said:
Maybe the click sound is due to ground current flowing between Teensy and whatever other hardware you have connected to Teensy?
Click to expand...
Hi Paul, thank you for the comments.

Only a mic is connected to the Teensy, no other hardware. And mic's ground goes to Teensy's ground.
I did run Teensy from a battery, its ground too is connected to Teensy, click behavior was the same. Having a completely isolated system changes nothing.
Adding a capacitor directly to SD power supply pin might help, but that pin is not accessible.
My humble feeling is it's not a ground loop since click period is very, very steady, being 640 KB of data. It seems more likely related to SD cluster search or so.
 
joepasquariello said:
You say you are writing 2048 bytes at a time. That is a multiple of 512, but have you tried 512? It MUST be 512 to avoid the long SD blocks.
Click to expand...
As evidenced by Bill Greiman, there is little the Teensy (any MCU) can do about activities on microSD card. Modern cards do use MUCH larger write-block-sizes. It is the card internal controller that decides what to cache and when to write. Writing single blocks only slows down the transfer speed.
 
YasarYY said:
Adding a capacitor directly to SD power supply pin might help, but that pin is not accessible.
Click to expand...
Even if you have a T4.1, I would try to write to an external uSD card. I know it is slower,but allows you to experiment with seperated power supply.
 
WMXZ said:
As evidenced by Bill Greiman, there is little the Teensy (any MCU) can do about activities on microSD card. Modern cards do use MUCH larger write-block-sizes. It is the card internal controller that decides what to cache and when to write. Writing single blocks only slows down the transfer speed.
Click to expand...
Writing one sector (512 bytes) at a time is necessary to avoid CPU blocking in the SD write. I agree that it won't yield the highest possible data rate to the SD. The OP's data rate is < 1 MB/s, which is not very high. I may be completely off-base in terms of whether the periodic click is caused by blocking, or is entirely related to power.
 
joepasquariello said:
Writing one sector (512 bytes) at a time is necessary to avoid CPU blocking in the SD write.
Click to expand...
A good data acquisition and logging system will have all processing at interrupt level and keep SD access at loop() level. This way waiting for SD write to return, does not matter. In particular, it does not interfere with acquisition that is handled at interrupt level. This is the beauty of ARM processors and maybe other modern architecture.
 
YasarYY said:
I acquire the audio from the ADC input
Click to expand...
You haven't said, but is this the Teensy ADC? This seems to be a poor choice for high-quality audio ... as you're discovering. Even if you fix the clicks, it'll be subject to signifcant digital noise from the processor itself.

MarkT said:
SDcard pull heavy currents when writing and erasing
Click to expand...
This is true, and unavoidable no matter how you schedule SD writes and how big they are. Teensy 4.1 is simply not big enough to have the hardware to mitigate against that.

It's also an issue if you use the SD card socket on the audio adaptor when recording audio from the adaptor itself - the layout and decoupling are insufficient, and you get buzzing or clicks due to SD writes. In that case, however, it can be mitigated, by using the SD socket on the Teensy 4.1. You're out of luck if you use a Teensy 4.0.

WMXZ said:
It is the card internal controller that decides what to cache and when to write
Click to expand...
YasarYY said:
This sentence describes the essence of the cause of this issue 😌
Click to expand...
I disagree - see the previous point.

YasarYY said:
I tried all of them, and below is the summary of the test journey.

* Separate ADC board
It is deemed overkill as a fix for periodic clicks

* Feeding Teensy 4.0 separately from the Adafruit microSD breakout
I'm using Teensy 4.1 which has an on-board Micro SD Socket.
Click to expand...
So ... you didn't "[try] all of them".
  • If you've genuinely tried everything else, and it hasn't worked, then an outboard ADC isn't "overkill", it's the only solution.
  • If you use an SD breakout then you have a chance to improve the isolation of the ADC and SD power routing, which is the issue at the heart of the clicks. It's not ideal, as it's extra hardware and not as fast as the onboard socket, but it may do the job, and probably more easily than an outboard ADC.
 
WMXZ said:
A good data acquisition and logging system will have all processing at interrupt level and keep SD access at loop() level. This way waiting for SD write to return, does not matter. In particular, it does not interfere with acquisition that is handled at interrupt level. This is the beauty of ARM processors and maybe other modern architecture.
Click to expand...
Good is not absolute. It’s a function of your requirements. Some applications have to do more than data acquisition and logging, and may need to do more in loop(). I don’t see any reason to have loop() spend 40 ms waiting for SD when there is an easy way to avoid it. That allows interrupts for critical operations and also have loop() available for tasks that are less critical, but can’t afford to wait 40 ms.
 
WMXZ said:
It is the card internal controller that decides what to cache and when to write.
Click to expand...
No. If you issue a write command, the card will write the data. Mostly. I see in version 9 of the specification a mention of cache. With a flush cache command that could take up to one second to complete. Yuck. So maybe use an older card to avoid that. I could never figure out what to do with all of the space on a 128MB card. :)

If a multi-sector write command is used, then it could delay writing sectors until all have arrived.

A problem here is that the SD card is at the bottom of the stack and SdFaT sits in the middle. Doing its own thing. One of those is to use multi-sector write commands even when you write a single sector of data. Holding that command open until the next call to sdFAT. If it is more sequential data, it continues the write command. If not, it completes the hanging write command before doing what was called for.

Figuring out what SdFat is doing and even customizing it is possible since it is open source.
 
h4yn0nnym0u5e said:
You haven't said, but is this the Teensy ADC? This seems to be a poor choice for high-quality audio ... as you're discovering. Even if you fix the clicks, it'll be subject to signifcant digital noise from the processor itself.
Click to expand...
Yes, it's the Teensy ADC. A choice being poor or good depends on the case. For my case, Teensy ADC is enough. I agree that I should've indicated my assessments are for my project and are not generalizable.

h4yn0nnym0u5e said:
an outboard ADC isn't "overkill", it's the only solution
Click to expand...
Again, it's overkill for my case for now.

h4yn0nnym0u5e said:
use an SD breakout then you have a chance to improve the isolation
Click to expand...
This will almost definitely solve the problem, or at least make the problem solvable (by means of power separation and/or caps).

So, what I can see from this thread and suggestions from other forums, there are 3 solutions and 1 workaround:
* Use an external SD breakout
* Use an external ADC
* Solder and use PSRAM chips: write to PSRAM from ADC, and after the acquisition completes, write to SD card from PSRAM.
* If new hardware is not wanted, patch samples at the software.
 
I saw that Greiman's post does not seem to suffer ticks with the 4.1

I am using my 4.0 boards and Adafruit SDIO board with ADC0. With them I see a VREF-related (?) glitch every 256 samples, which is the 512 byte write points. It is erratic for 28 samples at 815ksamp/s, droops then rises ~42mV (52 counts) until the next block write, there are also always 2 longer erratic blocks near the start:
blocks.png
Here, the top plot is stable ambient light return from a photodiode, and the bottom is with the ADC0 tied to 3.3:
Clipboard01.jpg

This is the effect on real-world high speed photodiode data, 0-3.3V:
Figure_1.png
From my mods of Bill Greiman's data logger, attached.

I'll test some power-smoothing capacitor addition myself next, and maybe use separate power sources.
 

Attachments

  • adc_v1.ino
    10 KB · Views: 10
BTW this is my Python viewer code I keep on the uSD card:
Code: 
import struct
import matplotlib.pyplot as plt


format_string = 'H'
dl = struct.calcsize(format_string)
l = []
with open('IsrLoggerTest.bin', 'rb') as file:
    while True:
        raw_data = file.read(dl)
        if raw_data!=b'':
            #print(f" {raw_data}", end='' )
            unpacked_data = struct.unpack(format_string, raw_data)[0]
            #print(f" {unpacked_data}", end='' )
            l.append(unpacked_data)
        else:
            print(f"{file.tell()}")
            break
ln = len(l)
print(f"len l {ln}")


#plt.plot(l[0:100000], 'o-r')
plt.plot(l, 'r')
plt.ylabel('12 bits')
plt.show()
 
You must log in or register to reply here.
Back
Top