Circular_Buffer

Tonton81,

OK, thanks. Maybe functions that only operate on circular arrays could have a '_array' appended to their name, like 'remove_array'?

Frank
 
Some features are specific to arrays, some to buffers, most work on both. It wouldn't be ideal to have seaparate names when there are alot of features to be handled would increase the function list, as long as it is, and future support for alternate buffer/array would require deprecating functions in terms of dual buffering support in a single function rather than by two different ones
 
Hello,

I try to use CB in my project to replace a "running average" but I have trouble to declare the circular buffer. I have several buffer that *may* have several size. The size in know at the init and belong to a structure. So doing this :

Code:
  for (int i = 0; i < nbTK; i++) {
    Circular_Buffer<float, infoTK[i].echantillonnage> coldJunction_CB[i];
    Circular_Buffer<float, infoTK[i].echantillonnage> compensedTemp_CB[i];
  }

I guess I have to create CB with a static size (ie : 32) THEN change the size with the relevant value ?

Thank you.
 
Yes the constructors must be defined at compile time, not during runtime, only the objects should be in your *for loop*
 
No, templates are created during compile time, size is set as is.
Is there a reason you want dynamic size?

For the for loop you posted, since you want to make it dynamic, you can probably just use a normal array for that. If you plan to hold values to use the circular buffer features it can't be dynamic sized
 
I intend to use circular_buffer as a moving average, and this average is something that the users may change to meet is needs. The number of sample to average the value is stored in EEPROM and is read at startup. This value can be changed using a serial command.
Not a big problem, I'll use another way to do this ;-)
Thank you
 
A function could be added if thats the case, but you need to set a max value for the size of the buffer, and a function made to read only the average of the X last values of it. It's a little tedious to make but it is possible.
 
@tonton81 - I have started experimenting with your circular_buffer library. I spent the day learning how to use it with MSC using your FlexCAN_t4 library as a guide for implementing it. There is a lot to learn:) I have MSC working with EventResponder now thanks to @KutrE. Will probably be asking questions here and there. What I am shooting for is to minimize the blocking effects of MSC and be able to queue up sector reads and writes to be processed in some way in the background queueing multi sector reads and writes only.

Each read or write transfer is comprised of three stages.

1) First stage is queuing and sending a CBW (Control Block Wrapper) for a multi sector read or write to a USB drive.

2) Second stage is queuing and sending a read or write data command to the USB drive.

3) The last stage is is the status stage. This is where the transfer is complete.

At first I was triggering the event for every stage of the transfer. But that was completely wrong. Really all I needed was to to trigger an event on completion of the transfer (Status Stage). I want to be able to queue each transfer and return to the test sketch and let the EventResponder dispatch any unsent queued transfers. I am still not sure if this is possible with the EventResponder. My test sketch is only doing sector reads and writes to a USB Mass Storage device. Keeping it low level and with the writes destroying any formatting on the drive:)

After some more experimenting with the EventResponder and circular_buffer libraries I hope to submit this version of MSC with just the driver for testing.
Am I going in the right direction?

Thanks
 
I havn't played with SD or USB utils so I wouldn't be of much help on that part, but a circular queue you could assign 512 bytes and when it is full via size() you can write that block to the SD or usb I guess. you are limited by the memory obviously and buffers are in powers of 2. You could also manage it with a fixed array in code, if its just appending bytes to an index, but you'll have to manage the indice counter whereas push or popping the queue shifts it in or out, a little less work for you.
 
I havn't played with SD or USB utils so I wouldn't be of much help on that part, but a circular queue you could assign 512 bytes and when it is full via size() you can write that block to the SD or usb I guess. you are limited by the memory obviously and buffers are in powers of 2. You could also manage it with a fixed array in code, if its just appending bytes to an index, but you'll have to manage the indice counter whereas push or popping the queue shifts it in or out, a little less work for you.

Thanks for the response. I actually have it working with one transfer just to test. I am using this structure for each data read or write.
Code:
	typedef struct {
		uint8_t type = NO_RD_WR; // Used for CMD_RD_10 or CMD_WR_10 only, else 0.
		bool completed;
		void *buffer;
		msCommandBlockWrapper_t *CBW;
	} MSC_transfer_t;

In the msReadSector() function:
Code:
// ---------------------------------------------------------------------------------------------------------
// Read Sectors using queued reads
// ---------------------------------------------------------------------------------------------------------
uint8_t msController::msReadSectors(void *sectorBuffer, const uint32_t BlockAddress, const uint16_t Blocks) {
	uint8_t msResult = 0;
#ifdef DBGprint
	Serial.printf("msReadSectors()\n");
#endif

	uint8_t BlockHi = (Blocks >> 8) & 0xFF;
	uint8_t BlockLo = Blocks & 0xFF;
	msCommandBlockWrapper_t CommandBlockWrapper = (msCommandBlockWrapper_t)
	{
	
		.Signature          = CBW_SIGNATURE,
		.Tag                = ++CBWTag,
		.TransferLength     = (uint32_t)(Blocks * (uint16_t)512),
		.Flags              = CMD_DIR_DATA_IN,
		.LUN                = currentLUN,
		.CommandLength      = 10,
		.CommandData        = {CMD_RD_10, 0x00,
							  (uint8_t)(BlockAddress >> 24),
							  (uint8_t)(BlockAddress >> 16),
							  (uint8_t)(BlockAddress >> 8),
							  (uint8_t)(BlockAddress & 0xFF),
							   0x00, BlockHi, BlockLo, 0x00}
	};
	mscTransfer.CBW = &CommandBlockWrapper;
	mscTransfer.buffer = sectorBuffer;
	mscTransfer.type = CMD_RD_10;	
[COLOR="#FF0000"]	uint8_t buf[sizeof(MSC_transfer_t)];
	memmove(buf, &mscTransfer, sizeof(mscTransfer));
	cbamsc.push_back(buf, sizeof(MSC_transfer_t));
cbamsc.list();
	Serial.printf("cbamsc.size() = %d\n", cbamsc.size());
	cbamsc.pop_front(buf, sizeof(MSC_transfer_t));
	memmove(&mscTransfer, buf, sizeof(mscTransfer));
	Serial.printf("cbamsc.size() = %d\n", cbamsc.size());[/COLOR]
//	msResult = msDispatchTransfer(mscXferEvent);
	msResult = msDoCommand(mscTransfer.CBW, mscTransfer.buffer);
//#ifdef DBGprint
	Serial.printf("transferIndex = %d\n",transferIndex);
	Serial.printf("mscTransfer.completed = %d\n"
					,mscTransfer.completed);
//#endif
	return msResult;
}

I zeroed out the mscTransfer stuct after I performed the push_back() function. I checked all of the elements in the mscTransfer to make sure they were zeroed out and then performed the pop_front() function and sent the struct to msDoCommand(). It read back 131072 bytes at 14Mb/s which was a 65536 byte 8bit buffer with two reads. So I think I am on the right track initially with your circular_buffer library.

Here is the instance of CB I used:
Code:
Circular_Buffer<uint8_t, MAX_TRANSFERS,sizeof(msController::MSC_transfer_t)> cbamsc;

After testing with just reading from the USB drive (thumb drive) I will create two instances of CB, one for sector reads and one sector write.

Nice work Tony:)

Edit: Output was:
Code:
Teensy USB MSD testing

Checking for msDrive1
msDrive1 init: OK

msDrive1 is NOT Mounted

   connected 1
   initialized 1
   USB Vendor ID: 0951
  USB Product ID: 1666
      HUB Number: 0
        HUB Port: 0
  Device Address: 1
Removable Device: YES
        VendorID: Kingston
       ProductID: DataTraveler 3.0
      RevisionID:     
         Version: 6
    Sector Count: 30218841
     Sector size: 512
   Disk Capacity: 15472046592 Bytes


Select:
   1) Test Sector Read Speed Drive 1
   3) Test Sector Write Speed Drive 1 ******** CAUTION!!! DESTROY'S FORMATTING OF DRIVE!! *********
Reading From USB Drive 1

Circular Array Buffer Queue Size: 1 / 10

First Entry: 40    0    0    0    32    0    0    0    144    255    5    32    (12 entries.)
Last Entry:  40    0    0    0    32    0    0    0    144    255    5    32    (12 entries.)

[Indice]      [Entries]

    0		40	0	0	0	32	0	0	0	144	255	5	32	(12 entries.)
cbamsc.size() = 1
cbamsc.size() = 0
transferIndex = 0
mscTransfer.completed = 1

Circular Array Buffer Queue Size: 1 / 10

First Entry: 40    1    0    0    32    0    0    0    144    255    5    32    (12 entries.)
Last Entry:  40    1    0    0    32    0    0    0    144    255    5    32    (12 entries.)

[Indice]      [Entries]

    1		40	1	0	0	32	0	0	0	144	255	5	32	(12 entries.)
cbamsc.size() = 1
cbamsc.size() = 0
transferIndex = 0
mscTransfer.completed = 1

Read 131072 bytes in 0.009599 seconds. Speed: 13.654756 MB/s
sizeof(SecBuf[0]) = 1
SecBufSize = 65536
----------------------------------------------------------
 
Last edited:
If I push back and pop front what exactly happens in terms of order of oldest vs newest data inside the buffer? I'm having a hard time understanding this concept.

Thanks
 
pushing back puts your data at the end. popping front pulls data from the front (oldest)
this is why you can do FIFO/LIFO
 
pushing back puts your data at the end. popping front pulls data from the front (oldest)
this is why you can do FIFO/LIFO

This is very confusing. What does put data at the end mean? In which direction does the data flow when you push back vs push front and pop front/back? Is there a pictorial representation somewhere? Very hard to explain this in words because there are multiple ways to view it.
 
1 2 3 4 5 6 7 8
push front 9 will become:
9 1 2 3 4 5 6 7 8
push back 6 will become:
9 1 2 3 4 5 6 7 8 6
pop front returns 9, your data becomes:
1 2 3 4 5 6 7 8 6
pop back returns 6, your data becomes:
1 2 3 4 5 6 7 8
 
OK what am I doing wrong?

Code:
#include "circular_buffer.h"

Circular_Buffer<float, 8> k1;


void setup() {
  k1.push_front(1.0f);
  k1.push_front(2.0f);
  k1.push_front(3.0f);
  k1.push_front(4.0f);
  k1.push_front(5.0f);
  k1.push_front(6.0f);
  k1.push_front(7.0f);
  k1.push_front(8.0f);
  k1.push_front(9.0f);

  //uint32_t temp = k1.pop_back();
  Serial.println(k1.pop_back());
  Serial.println(k1.pop_back());
  Serial.println(k1.pop_back());
  
  
  k1.list();
  
  
}

void loop() {
  // put your main code here, to run repeatedly:

}

This prints the following:

Code:
9.00

2.00

3.00


Circular Ring Buffer Queue Size: 5 / 8


Indice:  	[7]      	[0]      	[1]      	[2]      	[3]      	
Entries:	9.00		8.00		7.00		6.00		5.00

I'm trying to print the oldest values in the queue but the first value is always the newest value that was put inside the QUEUE.

The list() seems to display the correct values but as you can see the value 9.00 is printed first foolowed by 2.00 and 3.00 but I'm expecting 2.00, 3.00 and 4.00 in that order.
 
It seems that buffers smaller than 16 will not work properly when pop_back is used. So 8 or 4 won't work. However pop_front seems to work as it should. Or at least if I'm using them correctly.
 
Last edited:
can you setup a small example i can load to test?

The code snippet I posted is an example. You can run it. It just loads 1.0 to 9.0 and prints the last three values only the wrong values get printed. After that it prints what is left in the queue using the list function.


There is something wrong with the pop_back function when you set the buffer size to lower than 16. 16 or more works as it should.

I should add that I’m using the teensy 4 if it matters.
 
Last edited:
Fixed:

Code:
[URL="https://github.com/tonton81/Circular_Buffer/commit/6ec0bc149ac4b40a71b9b4aec26ccb11c34da917#diff-05031bd28244fb1791d9704abd4df93cb9b7b7f89a9e373b7a857ca91373bb87"]https://github.com/tonton81/Circular_Buffer/commit/6ec0bc149ac4b40a71b9b4aec26ccb11c34da917#diff-05031bd28244fb1791d9704abd4df93cb9b7b7f89a9e373b7a857ca91373bb87[/URL]

Code:
2.00

3.00

4.00


Circular Ring Buffer Queue Size: 5 / 8


Indice:  	[7]      	[0]      	[1]      	[2]      	[3]      	
Entries:	9.00		8.00		7.00		6.00		5.00
 
Fixed:

Code:
[URL="https://github.com/tonton81/Circular_Buffer/commit/6ec0bc149ac4b40a71b9b4aec26ccb11c34da917#diff-05031bd28244fb1791d9704abd4df93cb9b7b7f89a9e373b7a857ca91373bb87"]https://github.com/tonton81/Circular_Buffer/commit/6ec0bc149ac4b40a71b9b4aec26ccb11c34da917#diff-05031bd28244fb1791d9704abd4df93cb9b7b7f89a9e373b7a857ca91373bb87[/URL]

Code:
2.00

3.00

4.00


Circular Ring Buffer Queue Size: 5 / 8


Indice:  	[7]      	[0]      	[1]      	[2]      	[3]      	
Entries:	9.00		8.00		7.00		6.00		5.00


Great! :)

What was the problem?

Was it a bug I discovered? :D
 
yes, the queue itself worked, the return was wrong by one offset. essentially when deleting items from front or back, the head moves forward and tail moves backward, respectively. even though it does, the data is still there until it is overwritten. just the reading offset was off an indice
 
Back
Top