Full motion video + sound on the T4 + ILI9341. Eliminating tearing with ILI9341_t3n..

Status
Not open for further replies.

Mike Chambers

Well-known member
I'm working on a library that allows to you play full motion video with sound on a Teensy 4.x with an ILI9341. I'm not sure how useful such a library would be, but it seemed like a fun project to work on.

It's essentially working. I'm using Kurt's awesome ILI9341_t3n library with DMA mode. I found I have to enable continuous async update mode. If I do manual updates when the framebuffer has been completely updated, I'm only able to squeak out about 10 FPS or less for some reason.

Am I just going to have to stay in continuous mode, keep double framebuffers and only update the DMA referenced one after checking asyncUpdateActive() for false? Or is there a better way where I can call for an update manually while getting a faster refresh rate?

Tech details about the library... it's pretty simple, I just made an encapsulation format I call JVI (Jpeg VIdeo) that contains a header with some info and then interleaves JPEG file and raw audio data, plus a frame index at the end to make quick frame seeks possible. VirtualDub2 makes it easy to export proper JPEG sequences along with a raw audio file, which you would then run through a small PC utility to turn them into a JVI file that gets thrown on your SD card.

Here you can see it working, with the tearing evident. You'll have to turn up your volume, it's just a little PC speaker connected.

 
Last edited:
Pretty cool! I never would, have thought that would be possible. But it makes my head hurt just thinking about it.

-jim lee
 
Yeah, it's amazing what these modern micros can do. In the mid-late 90's this would have been considered powerful even for a desktop if it just had more RAM.

My thought was it could be useful for someone who wants slick startup animations, or animations to play while the system is doing some processing in the background, or whatever else like that on their projects.

File sizes are reasonable too. That 45 second clip is only 13 MB. Watched a full 45 minute Star Trek episode on it too to check for any hiccups on large videos, it was about 850 MB. So you could easily fit small animations or clips in the onboard flash without even needing an SD card.
 
This is cool. For my msx2 emulator, I'm using modified ili9341_t3n to update specific part of the screen only. I update one of the 10x10 tiles of the screen at a time when any of the tile gets changed. I could achieve 60fps with no noticeable tearing in games but movie file would be a totally different beast, I guess. Perhaps using appropriate similarity measure of the pixel change can help.
Did you use external DAC for the sound? I don't think T4.1 has internal DAC like T3.6....
 
This is cool. For my msx2 emulator, I'm using modified ili9341_t3n to update specific part of the screen only. I update one of the 10x10 tiles of the screen at a time when any of the tile gets changed. I could achieve 60fps with no noticeable tearing in games but movie file would be a totally different beast, I guess. Perhaps using appropriate similarity measure of the pixel change can help.
Did you use external DAC for the sound? I don't think T4.1 has internal DAC like T3.6....

The Teensy 4.0 and 4.1 does not have internal DAC's. Sound output options include:
  • Use the main digital I2S bus (which the revision D audio shield uses) -- either using the audio shield, or using another I2S output device (such as the PT8211 kit PJRC sells, or other I2S devices);
  • Use the second digital I2S bus with an external I2S device;
  • Use the MQSL/MQSR pins -- note these pins overlap with pins 10 and 12 which are used by the first SPI bus, so you likely would need to use the second I2C bus to drive the display;
  • There are S/PDIF in/out pins -- I don't know the level of support;
  • I would imagine MIDI and USB audio outputs might be able to be used.
 
This is entirely possible, and a library would be very useful I think!

I'm currently getting 30+ FPS video and sound streamed from an SD card (SdFat Beta, SDIO, clock 70Khz) using a Teensy 4.0, ILI9342 driving a 320x240 display through a slightly modified ILI9341_t3 library, with SPI set to 75MHz. Video is pre-encoded using a Processing script I found and have forgotten the origins of (it may even be PJRC), but it basically creates an output file of consecutive 320x240 16bpp frame info followed by audio info, so it's a simple matter to loop through the file, read in each frames video and audio chunk into a buffer and push out over SPI and I2S. There is horizontal tearing visible when there is high motion in the video, as the transfer speed is still not within the displays refresh rate of 60Hz, I think. With the Teensy at 600Mhz and the aforementioned speed settings, the frame read time is ~5400uS and the buffer transfer time over SPI is between 15500-20500uS, depending on output resolution (320x180 or 320x240).

I encoded the same clip as you did, here's a video of it playing. Towards the end of this video, I rewind/fast forward through the clip, and you can see the tearing there:

 
This is entirely possible, and a library would be very useful I think!

I'm currently getting 30+ FPS video and sound streamed from an SD card (SdFat Beta, SDIO, clock 70Khz) using a Teensy 4.0, ILI9342 driving a 320x240 display through a slightly modified ILI9341_t3 library, with SPI set to 75MHz. Video is pre-encoded using a Processing script I found and have forgotten the origins of (it may even be PJRC)

Wow, nice, almost like the 3.6 which does the same and has simultanous audio ;)

On the NXp website, there is an example for h.264 Video.
Has anyone tried to port it?
 
That is cool. This works for both of these I assume?

https://www.pjrc.com/store/display_ili9341.html - discontinued
https://www.pjrc.com/store/display_ili9341_touch.html

I just ordered the latter out of a vague notion of wanting some debug display option on hand.

That second link is the one I'm using. I've also tried a random Chinese 3.2" I got on Amazon and it works fine too. Any with the ILI9341 should theoretically work.

This is cool. For my msx2 emulator, I'm using modified ili9341_t3n to update specific part of the screen only. I update one of the 10x10 tiles of the screen at a time when any of the tile gets changed. I could achieve 60fps with no noticeable tearing in games but movie file would be a totally different beast, I guess. Perhaps using appropriate similarity measure of the pixel change can help.
Did you use external DAC for the sound? I don't think T4.1 has internal DAC like T3.6....

Is your msx2 emulator posted anywhere? I'd be interested in seeing that, I love emulation. For the sound, I'm just using simple PWM analogWrite() but an external DAC would obviously be better.

This is entirely possible, and a library would be very useful I think!

I'm currently getting 30+ FPS video and sound streamed from an SD card (SdFat Beta, SDIO, clock 70Khz) using a Teensy 4.0, ILI9342 driving a 320x240 display through a slightly modified ILI9341_t3 library, with SPI set to 75MHz. Video is pre-encoded using a Processing script I found and have forgotten the origins of (it may even be PJRC), but it basically creates an output file of consecutive 320x240 16bpp frame info followed by audio info, so it's a simple matter to loop through the file, read in each frames video and audio chunk into a buffer and push out over SPI and I2S. There is horizontal tearing visible when there is high motion in the video, as the transfer speed is still not within the displays refresh rate of 60Hz, I think. With the Teensy at 600Mhz and the aforementioned speed settings, the frame read time is ~5400uS and the buffer transfer time over SPI is between 15500-20500uS, depending on output resolution (320x180 or 320x240).

I encoded the same clip as you did, here's a video of it playing. Towards the end of this video, I rewind/fast forward through the clip, and you can see the tearing there:


That looks great, and I like your little menu system at the beginning. Your tearing is less intrusive than mine. Sounds like you are quicker with how long you're messing with the framebuffer memory with those raw frames. You can just do a quick one-off memory copy, where I'm having to decode one JPEG MCU block at a time, write that into the framebuffer then process the next one. I guess making the likelihood higher of me being in the middle of updating things when the display library does it's DMA copy. Double-buffering could fix that... at the cost of more RAM.
 
Last edited:
All three S/PDIF Outputs work, can work simultanous -- the S/PDIF input exists in two variants (one simple 44.1kHz only OR the other with resampling).

I thought they were now working, but I did not know for certain, since I haven't used them.

Is there a simple setup that would allow using pins 14/15 on the Teensy 4.0/4.1 to play audio? My main interest is small portable battery driven units to generally play pre-recorded sounds. Yeah, I already have several other solutions as I mentioned above, but it is always good to have other options.
 
I thought they were now working, but I did not know for certain, since I haven't used them.

Is there a simple setup that would allow using pins 14/15 on the Teensy 4.0/4.1 to play audio? My main interest is small portable battery driven units to generally play pre-recorded sounds. Yeah, I already have several other solutions as I mentioned above, but it is always good to have other options.

Paul can answer this better - I've tested with 100uF capacitors per pin - between amplifier and pin.
Not sure if this is the best way - I'm a analog electronic noob. Edit :Opps sorry that was MQS on 10/12

For S/PDIF you can get cheap D/A from Amazon, EBay or ALIE - But they need an additional amplifier.
 
That looks great, and I like your little menu system at the beginning. Your tearing is less intrusive than mine. Sounds like you are quicker with how long you're messing with the framebuffer memory with those raw frames. You can just do a quick one-off memory copy, where I'm having to decode one JPEG MCU block at a time, write that into the framebuffer then process the next one. I guess making the likelihood higher of me being in the middle of updating things when the display library does it's DMA copy. Double-buffering could fix that... at the cost of more RAM.

Thanks, I'm using LittleVGL for the menu system, it's really nice! And yes, because I pre-encode the video file, it's literally just a read directly into the memory buffer and then a SPI transfer of that buffer. Speed at the cost of file size and flexibility, it's a few steps to prepare the video file before hand :) A library that can decode native files on the fly would be very useful!
 
My video files still need to be encoded into my custom format, but since it's utilizing JPEG encoding, they come out pretty small at least.

Maybe making it able to support an already widely-used format would be better. Like MKV. I'm not sure how complicated that would be to implement, but then you could just encode MJPEG videos with existing tools like ffmpeg. I almost wonder how difficult it would be to implement MPEG-1/2 decoding? The CPU power is there for 320x240 but I'm not sure what the memory requirements would be. Certainly H264 is a bit heavy for a Teensy.

It seems like a lot of effort for little gain though. It's still easy enough to export JPEG sequences in vdub and run them through my utility. :)
 
Paul can answer this better - I've tested with 100uF capacitors per pin - between amplifier and pin.
Not sure if this is the best way - I'm a analog electronic noob. Edit :Opps sorry that was MQS on 10/12

For S/PDIF you can get cheap D/A from Amazon, EBay or ALIE - But they need an additional amplifier.
Hmmm, I found this link from Paul in 2015 mentioning your work:

Lets see, there is:

I don't know if any of those are safe to use at TTL voltage levels, or whether you need a voltage level converter to bring the S/PDIF output down to 0.6 volts. And as you said, you need amplification as well. Fortunately I have some Adafruit speakers for portable usage that includes amplification:
 
Is your msx2 emulator posted anywhere? I'd be interested in seeing that, I love emulation. For the sound, I'm just using simple PWM analogWrite() but an external DAC would obviously be better.

Why I haven't thought about PWM? I will start working on enabling sound system right now :)
I'm basically porting blueMSX. It's a nice emulator but it uses a lot of malloc (almost unnecessarily) internally so I need to take care of them. Porting is still work-in-progress but if you want I'm happy to open my github for you to look/test around. (I'm currently embedding roms in the source code so I can't make the code public until I get read of them from the repository.)
Also I'm trying add a physical slot so that I can play actual cartridges. They are 5.0V TTL so it actually works pretty well with 3.3v CMOS as long as that you take care of 5V->3.3V singal with some 5V-tolerant buffers (to interface data bus I'm using 74lvc245.) My working setup already has 512kB parallel sram so adding cartridge support should not be too hard. (blueMSX maintains ~200kB of video ram on top of the framebuffer so I needed additional memory anyway.)
https://photos.app.goo.gl/8u382J7SmoQvP86T9
 
Last edited:
Why I haven't thought about PWM? I will start working on enabling sound system right now :)
I'm basically porting blueMSX. It's a nice emulator but it uses a lot of malloc (almost unnecessarily) internally so I need to take care of them. Porting is still work-in-progress but if you want I'm happy to open my github for you to look/test around. (I'm currently embedding roms in the source code so I can't make the code public until I get read of them from the repository.)
Also I'm trying add a physical slot so that I can play actual cartridges. They are 5.0V TTL so it actually works pretty well with 3.3v CMOS as long as that you take care of 5V->3.3V singal with some 5V-tolerant buffers (to interface data bus I'm using 74lvc245.) My working setup already has 512kB parallel sram so adding cartridge support should not be too hard. (blueMSX maintains ~200kB of video ram on top of the framebuffer so I needed additional memory anyway.)
https://photos.app.goo.gl/8u382J7SmoQvP86T9

This is seriously cool! Neat breadboard work, too. The physical slot would be a good addition, would make it feel a little more authentic. :)

My GitHub username is mikechambers84.

If you're interested, I have some PS/2 keyboard interface code that I used in my Teensy PC emulator. Though USB may be a better choice these days. I had some PS/2 KB's around and the interface is simple enough so I used that.
 
This is seriously cool! Neat breadboard work, too. The physical slot would be a good addition, would make it feel a little more authentic. :)

My GitHub username is mikechambers84.

If you're interested, I have some PS/2 keyboard interface code that I used in my Teensy PC emulator. Though USB may be a better choice these days. I had some PS/2 KB's around and the interface is simple enough so I used that.

Invited. Note that I used PlatformIO to build the source, and only t40 branch works right now. I need to add information on wiring...
 
Status
Not open for further replies.
Back
Top