T4 Pixel Pipeline Library

[Rezo]

All fixed:
PXP_OUT_PS_LRC had to be set to ((479) << 16) | (319); (was 480, 320)
and
PXP_OUT_LRC has to be set to ((95) << 16) | (319); (was 96, 320)

 

[mborgerson]

All fixed:
PXP_OUT_PS_LRC had to be set to ((479) << 16) | (319); (was 480, 320)
and
PXP_OUT_LRC has to be set to ((95) << 16) | (319); (was 96, 320)

Yeah, I got so tired of making that mistake that I wrote a macro to remind myself that I'm setting a corner:

#define SetCorner(reg, h, v) reg = (h<<16 + v)

The macro is used like this:
SetCorner(PXP_OUT_PS_LRC, width-1,height-1);

I thought about doing the subtraction in the macro, but that would have serious consequences if you called it for an upper left corner:

SetCorner(PXP_OUT_PS_ULC, 0,0); // Not good if you subtract one inside macro

 

[mjs513]

Know this is more than a year later but started playing with the PXP library using a OV5640 camera. The basic setup is:
1. ILI9488 on SPI using ILI9488_t3 library
2. Framesize = 480x320 (HXGA) yes this works.
3. Framebuffers setup in the sketch as 480*320
4. Using @KurtE's and my Camera Library
5. Using Flexio3 on a Teensy Micromod

Sketch modified @Rezo sketch in post #27, severely modified at least for setup. Just testing Rotation to start.

To start PXP I am using:

void start_pxp(){
  PXP_init();
  Serial.printf("bwitdh: %d, bheight: %d\n", bwidth, bheight);
  SetCorner(PXP_OUT_PS_ULC, 0, 0);
  SetCorner(PXP_OUT_PS_LRC, (bwidth - 1), (bheight - 1));

  PXP_input_buffer(s_fb, 2, bwidth, bheight);
  PXP_input_format(PXP_RGB565);
  PXP_output_buffer(d_fb, 2, bheight, bwidth);
  PXP_output_format(PXP_RGB565);
}

Rotation I made generic

void pxp_rotation(int r) {
  if(!pxpStarted) {
    Serial.println("You forgot to start PXP, use 's' to start.....");
  } else {
    capture_frame(false);
    PXP_rotate(r);
    PXP_process();
    draw_frame();
  }
}

Draw frame is pretty straigth forward

void draw_frame() {
  //arm_dcache_flush(d_fb, sizeof(d_fb)); // always flush cache after writing to DMAMEM variable that will be accessed by DMA
  tft.writeRect(CENTER, CENTER, tft.width(), tft.height(), (uint16_t *)d_fb );
}

Frame grap is straight forward

camera.useDMA(true);
  camera.readFrame(s_fb , sizeof_s_fb);
  arm_dcache_flush((uint16_t*)s_fb, sizeof_s_fb); // always flush cache after writing to DMAMEM variable that will be accessed by DMA

The whole sketch is attached for reference. Problem in next post - this is getting long

 

[mjs513]

Here's the problem
For Rotation 0 - seems to be working but image is not centering as specified in the writeRect function

Rotation 1 - notice it draws it on the top have of the screen as what looks like a wrapped image:

Rotation 2 - see rotation 0 notes

As you might expect rotation 3 but it also leaves rotaton2 data in the buffer

I suspect its something to do with the LRC/ULC settings but its giving me a headache. Any help will be appreciated

Thanks

Mike

 

[mjs513]

Made a little progress - but out image is always a square. Kind of giving up on rotations - ps it easier to set the display to do the rotation if thats all you are doing on an image - probably not good to do it with video unless display doesn't have a hardware setting.

To get that I added a single command:

    memset((uint8_t *)d_fb, 0, sizeof_d_fb);
    PXP_rotate_position(1);  //Rotation position, 1=input  <------- added to the rotation on the input buffer
    capture_frame(false);
    PXP_rotate(r);
    PXP_process();
    draw_frame();

Cheers

 

[KurtE]

Sorry @mjs513, I am not much help on this one. I keep meaning to learn more about the PXP stuff. As there might be places it could come in handy.

For example with the ILI9488_t3 code, we typically store the pixels in 16 bit 565 format, but the SPI interface does not handle this, so we convert into a 24 bit 666 (or 888) format. We have code in place that when we write to the screen we convert each pixel.

An interesting case is for Asynchronous DMA updates. Currently the code uses two secondary DMA buffers, that I chain to each other which holds a certain number of converted pixels. Then start up the DMA, and when it finishes one buffer it interrupts and fills with the conversions for the next N pixels... Would be interesting to see how well the PXP conversion code maybe converts on the fly from 16 bit to 32 bit 888 mode which is passed to the SPI output register (who only outputs 24 bits of it... I am assuming they have a setup for handling this with DMA... But again I need to reread this section again (and again)...

 

[mjs513]

For example with the ILI9488_t3 code, we typically store the pixels in 16 bit 565 format, but the SPI interface does not handle this, so we convert into a 24 bit 666 (or 888) format. We have code in place that when we write to the screen we convert each pixel.

Not an expert on this by far - was torture just trying to get rotation semi working. I do know that you can set up the source input buffer as RG565 and then setup the out buffer to RGB888 and it should convert. No clue how to actually try it though and set up the out buffer for RGB888.

 

[Rezo]

I had a very had time with the rotation. But eventually got it working using a frame sized destination buffer and two 1/10th source buffers - not a standard implementation at all.

With SDRAM can go full sized frame buffers and let it do its thing.

@mjs513 try copy the destination buffers content to a file in littleFS and convert it to a bitmap on your computer to make sure the issue is the PXP and not the lcd driver, or the interface between the two.
To me it looks like a rotation issue on the display driver (from my days writing the FlexIO 8080 library)

 

[mjs513]

I had a very had time with the rotation. But eventually got it working using a frame sized destination buffer and two 1/10th source buffers - not a standard implementation at all.

With SDRAM can go full sized frame buffers and let it do its thing.

Thanks for joining the fray here. Right now I am using QVGA with a ILI9341 so using 2 full size buffers:

DMAMEM uint16_t s_fb[320*240] __attribute__ ((aligned (64)));
uint16_t d_fb[320*240] __attribute__ ((aligned (64)));

Right now I have just been taking guesses based on the contents of this thread on how things work and why. Wish there was a basic tutorial

For instance to start PXP I am using a function that looks like this:

PXP_init();
  Serial.printf("bwitdh: %d, bheight: %d\n", bwidth, bheight);
  SetCorner(PXP_OUT_PS_ULC, 0, 0);
  SetCorner(PXP_OUT_PS_LRC, (bwidth - 1), (bheight - 1));
  PXP_input_buffer(s_fb, 2, bwidth, bheight);
  PXP_input_format(PXP_RGB565);
  PXP_output_buffer(d_fb, 2, bheight, bwidth);
  PXP_output_format(PXP_RGB565);

PXP_OUT_PS_ULC and LRC is defined in the manual as:

PXP_OUT_LRC:
This register contains the size, or lower right coordinate, of the output buffer NOT rotated. It is implied that the upper left coordinate of the output surface is always [0,0]. When rotating the framebuffer, the PXP will automatically swap the X/Y, or WIDTH/ HEIGHT, to accomodate the rotated size.

PXP_OUT_PS_ULC
This register contains the upper left coordinate of the Processed Surface in the output frame buffer (in pixels).

So for the ILI9341 the lower is 320x240 or should it be 240x320??????? Note didn't help if I changed it, by that I mean still get a square of 240x240. Found that @mborgerson used a setCorner macro - lazy so am using it (SetCorner(reg, h, v) reg = (h<<16 | v))

Now PXP_input_format and output_format is pretty straightforward so that was the easy part.

But this is the part that confuses me - copied from one of your code snippets:

PXP_input_buffer(s_fb, 2, bwidth, bheight);
  PXP_output_buffer(d_fb, 2, bheight, bwidth);

Do not understand why you are switching width's and height between the two but it works to get a decent image if I make them match looks garbled. So diving depther the code for output_buffer:

next_pxp.OUT_BUF = buf;   <---------------------- assmuing ptr to buffer
  next_pxp.OUT_PITCH = PXP_PITCH(bytesPerPixel * width);     <--------------- bytes per output row????
  next_pxp.OUT_LRC = PXP_XCOORD(width-1) | PXP_YCOORD(height-1);  <--------------- just noticed this, so dont have to do in sketch......

So why the reverse in width and height between the 2?

Also what is the correct calling sequence for commands guessed and seems to work but I did notice another one:

//Call to make sure the last process is finished before using the buffer with a display
void PXP_finish();

should I be calling this before drawing the frame?

 

[mjs513]

@mjs513 try copy the destination buffers content to a file in littleFS and convert it to a bitmap on your computer to make sure the issue is the PXP and not the lcd driver, or the interface between the two.
To me it looks like a rotation issue on the display driver (from my days writing the FlexIO 8080 library)

Will give it a try - but question on lrc/ulc - do these change with rotation

 

[mjs513]

Got it !!! Found it by accident:

solution:

void pxp_rotation(int r) {
  if(!pxpStarted) {
    Serial.println("You forgot to start PXP, use 's' to start.....");
  } else {
    memset((uint8_t *)d_fb, 0, sizeof_d_fb);
    PXP_rotate_position(0);  //Rotation position, 1=input
    capture_frame(false);
    if(r == 0 || r == 2) {
      PXP_output_clip(bheight-1, bwidth-1);
    } else {
      PXP_output_clip( bwidth-1, bheight-1);
    }
    PXP_rotate(r);
    PXP_process();
    draw_frame();
  }
}

 

[Rezo]

So what was it?

 

[mjs513]

So what was it?

Have to swap the values for the OUT_LRC register - didn;t register (pardon the pun) that this was different than PXP_OUT_PS_LRC. PXP_OUT_LRC appears to set the display LRC, hence the need for:

if(r == 0 || r == 2) {
      PXP_output_clip(bheight-1, bwidth-1);
    } else {
      PXP_output_clip( bwidth-1, bheight-1);
    }

Once I did this everything fell into place to get portrait mode working at least for the 320x240 display.

 

[mjs513]

@KurtE - @Rezo
here is a working sketch using a OV5640 or OV2640. Tested with QVGA, QQVGA and 240x240 framesizes

 

[KurtE]

@mjs513 - glad you have it working! Will have to play with it some more.

(All) But I am wondering how/where I might use it on a normal Teensy, where I don't have external memory and I am not using eLCD?

That is are there examples on how one might use it to maybe output to a TFT display over SPI without having the need to have
a full-size output buffer?

The documentation shows some information about doing some of the stuff as blocks with the LCD subsystem, that it maybe
can do a 16 bit VGA size output with maybe only something like a 20 or was it 30K buffer. But talked about signals between
the two sub-systems.

Is it possible to do similar with output to SPI or the like? Are there examples?

Things like upsize or downsize and downsize an image and potentially rotate it 90 degrees?

Convert a RGB565 image into an RGB888 (32 bit) in pieces. Better yet, if it could direct the output to SPI... For example, with the ILI9488 display code, we typically store image in 16 bit format but output 24 bits to display. For DMA outputs to the screen I have to do this in chunks, where I use an interrupt at the end of each DMA output, to convert the next chunk into the 32 bit format. So far I am assuming that it is not possible to interject a conversion into a DMA operation. LIke: memory->PSP convert->SPI.

Convert a camera grayscale 8 bit image to RGB565 format... In this case it would be great if the source could be from SPI... My reading of the
document they have some Monochrome 8bit Y single plane... But I think that is different?

Sorry just having a hard time understanding how to use some of this.

Thanks

 

[Rezo]

@KurtE the partial output buffers only work with the eLCDIF handshake.

The only way to use this on a non SDRAM Teensy is to use one screen size source buffer and two partial destination buffers

I was able to rotate a landscape frame generated by LVGL with this method on a MicroMod.

LVGL wrote into the full frame buffer, and I then while rotating into one destination buffer, would transfer into the other.

You might be able to do it with a smaller source buffer, but it will be very complex.

There is no full DMA service to write directly to SPI or read from SPI into the PXP.

The only advantage here is you can offload the tasks from the main core. So while reading into one destination buffer it rotates the other and then sends it off to the display and vice versa.

You can see the code I posted here - https://forum.pjrc.com/index.php?threads/t4-pixel-pipeline-library.64860/post-320150

 

[KurtE]

@KurtE the partial output buffers only work with the eLCDIF handshake.

The only way to use this on a non SDRAM Teensy is to use one screen size source buffer and two partial destination buffers

I was able to rotate a landscape frame generated by LVGL with this method on a MicroMod.

LVGL wrote into the full frame buffer, and I then while rotating into one destination buffer, would transfer into the other.

You might be able to do it with a smaller source buffer, but it will be very complex.

There is no full DMA service to write directly to SPI or read from SPI into the PXP.

The only advantage here is you can offload the tasks from the main core. So while reading into one destination buffer it rotates the other and then sends it off to the display and vice versa.

You can see the code I posted here - https://forum.pjrc.com/index.php?threads/t4-pixel-pipeline-library.64860/post-320150

Thanks, will take a look.

At this point, I think I will defer for now. If the main issue is simply to rotate by 90, 180, 270 with most of our displays,
we can simply tell the display to set their rotation and the rest is taken care of.

If I find that it is not sufficient for some specific needs, like maybe I need to rotate the camera output by 90 degrees to be displayed
within a portion of the TFT screen, I would probably addsomething like: tft.writeRotatedRect(x, y, w, h, pixels, 1);
where the 1 is like the current rotations of screen...

We do some similar stuff in some of the displays that don't fully support rotations...

Thanks again

 

[Rezo]

The only real problem with on-screen rotation such as the ILI9341/9488 is that while we can rotate by changing the MADCTL register on the display, the actual pixels are still displayed in portrait mode, top to bottom (bottom being where the ribbon cable is), regardless of rotation settings. When optimizing for high frame rate, this causes visible tearing and cannot be avoided.
By rotating with the PXP before sending the data out to the display (keeping the MADCRL rotation at 0 degrees) you can avoid that tearing affect.

The real benefits of the PXP in my eyes are:

• Rotation of a buffer when using the eLCDIF
• 2D accelerations specifically for rotating objects, alpha blending and image scaling - mostly when using a GUI framework
• Color conversions

If it's none of the above, I don't see a good excuse to use it.

 

[mborgerson]

The only real problem with on-screen rotation such as the ILI9341/9488 is that while we can rotate by changing the MADCTL register on the display, the actual pixels are still displayed in portrait mode, top to bottom (bottom being where the ribbon cable is), regardless of rotation settings. When optimizing for high frame rate, this causes visible tearing and cannot be avoided.
By rotating with the PXP before sending the data out to the display (keeping the MADCRL rotation at 0 degrees) you can avoid that tearing affect.

The real benefits of the PXP in my eyes are:

• Rotation of a buffer when using the eLCDIF
• 2D accelerations specifically for rotating objects, alpha blending and image scaling - mostly when using a GUI framework
• Color conversions

If it's none of the above, I don't see a good excuse to use it.

The PXP can be very useful for some applications. A little over 3 years ago, I used it to manipulate images for a slide show app. The PXP made possible things like fade-in, fade-out, moving transitions, and Ken Burns effect (combination of translation and scaling). A search of the forum for "Slide Show" will find the post demonstrating the PXP-enabled effects as displayed on a PC host.

 

[KurtE]

The PXP can be very useful for some applications. A little over 3 years ago, I used it to manipulate images for a slide show app. The PXP made possible things like fade-in, fade-out, moving transitions, and Ken Burns effect (combination of translation and scaling). A search of the forum for "Slide Show" will find the post demonstrating the PXP-enabled effects as displayed on a PC host.

Thanks,

There are a lot of great stuff shown in your video!

This is an improved Slide Show display. The QVGA slide images and transitions were recorded with the Win10 screen recorder. The slide images and transitions were sent to the PC host application using the T4.1 Dual serial connection so a dedicated USB channel was available for the image data. At the same time, the slide show was sent to an ILI9341 display. Sending to both at the same time dis not seem to slow the display significantly. Ah, the wonders of well-written drivers using DMA!

Now that I'm not limited to the QVGA resolution of the ILI93341, I plan to replace the .BMP file with full VGA versions and see how long it takes to do the transitions and display with 4 times as many pixels.

At some point it will be interesting to play with some of these more advanced features!

 

[mjs513]

Ok all been at it some more with trying out scaling and rotation took most of yesterday to figure out.

Did find that just using a macro to set the corners never changed the actual registers so initially things were screwy, so added a function to my fork of the library: https://github.com/mjs513/T4_PXP/tree/pxp_t4_mods.

Set Corners function:

#define PXP_OUT_PS_X(x)  (((uint32_t)(((uint32_t)(x)) << 16)) & (0x3FFF0000U))
#define PXP_OUT_PS_Y(x)  ((uint32_t)((uint32_t)(x)) & (0x3FFFU))
void PXP_SetOutputCorners(uint16_t ulc_x, uint16_t ulc_y, uint16_t lrc_x, uint16_t lrc_y)
{
   next_pxp.OUT_PS_ULC = PXP_OUT_PS_Y(ulc_y) | PXP_OUT_PS_X(ulc_x);
   next_pxp.OUT_PS_LRC= PXP_OUT_PS_Y(lrc_y) | PXP_OUT_PS_X(lrc_x);
}

now it actually changing the registers.

The other issue was that when I tried to use the PXP_input_scale function it just produced a blank colored block but no image! So went back to the sdk and lifted they way they did but found when you tried to set the decimation factors the image failed to display. Commenting that out it worked. So the two functions added were:

void PXP_setScaling(uint16_t inputWidth, uint16_t inputHeight, uint16_t outputWidth, uint16_t outputHeight)
and
void PXP_GetScaleFactor(uint16_t inputDimension, uint16_t outputDimension, uint8_t *dec, uint32_t *scale)
which setScaling calls

so what you wind up with going from QVGA down to 160x120 (Sorry for the glare)

Now with rotation to landscape - still something off with that one:

Its cropping the image but not sure why.

I will push up the test case to the repo if you are interested

 

[Rezo]

Amazing work Mike!

Now get the touchscreen to work and try to scale in-out using a gesture

 

[mjs513]

Ok folks. While testing what I had with different size images than the flexio_teensy_mm found that there were several issues with rotation and scaling. So for the past few days been working on resolving those issues and finally found the solution.

In the latest iteration of the sketch everything is done from a single function for rotation/scaling/flipping. Now it seems to work better for a variety of images. I am posting my rough sketch - still have clean up to do but you should get the idea.

Once I do that will update the repo with sketch and probably post some images.

EDIT: Just as a note, I ran it on a T4.1 with the buffers in EXTMEM and didn't seem to be much of a performace hit - at least with 480x320 pixels.

 

[mjs513]

Quick update:
Pushed latest changes to repo: https://github.com/mjs513/T4_PXP/tree/pxp_t4_mods

fixed a minor bug when image_width > image_height.
added some comments.
theoretically you should just be able to drop the PXP functions into your sketch and it should work. You wlll need to udpate the draw_frame function to whatever display you are using if not one of the ones we already support.

Cheers. Think I am done with PXP now.

 

[KurtE]

Looks like it runs on my MMOD with ILI9488