mjs513
Senior Member+
Ok will stop playing on this end. I did put a T4.1 with 8Mb PSRAM but the 512, 1G, and 2G files failed to transfer any of the PDF files and essentially disconnected the flash drives from windows to where I had to power cycle.
MTP Responder Contribution
- Thread starter hubbe
- Start date
Ok will stop playing on this end. I did put a T4.1 with 8Mb PSRAM but the 512, 1G, and 2G files failed to transfer any of the PDF files and essentially disconnected the flash drives from windows to where I had to power cycle.
KurtE
Senior Member+
Totally understand.... Pretty frustrating.
This morning I decided to waste some time and see if going at this completely wrong. So tried a KISS version of the code...
Code:
bool MTPD::SendObject() {
bool return_value = false;
uint8_t print_count = 0;
pull_packet(rx_data_buffer);
read(0,0);
uint32_t len = ReadMTPHeader();
uint32_t index = sizeof(MTPHeader);
printf("MTPD::SendObject: len:%u\n", len);
elapsedMicros em_total = 0;
uint32_t cb_write = MTP_RX_SIZE - index; // how many data in usb-packet
if (cb_write > len) cb_write = len;
if(!storage_->write((char*)rx_data_buffer + index, cb_write)) goto abort;
len -= cb_write;
cb_write = MTP_RX_SIZE;
while (len)
{
printf(".");
if (!(++print_count & 0x3f)) printf("\n");
// Build in timeout...
uint32_t elapsed_before_packet = em_total;
elapsedMillis em_packet = 0;
while(!usb_mtp_available())
{
if (em_packet > 1000)
{
printf("\nTimeout: %u\n", elapsed_before_packet);
goto abort;
}
}
usb_mtp_recv(rx_data_buffer, 60);
if (cb_write > len) cb_write = len;
if(!storage_->write((char*)rx_data_buffer, cb_write)) goto abort;
len -= cb_write;
}
return_value = true;
printf("$");
abort:
storage_->close();
printf(">> %u\n", (uint32_t)em_total);
return return_value;
}First tried a simple smaller file:
Code:
CMD: 100d(SEND_OBJECT)l: 12 T:71
MTPD::SendObject: len:44082
................................................................
......................$>> 539871
RESP:2001(RSP:OK)l: 12 T:71Which worked...
Then tried a larger file
Code:
CMD: 100d(SEND_OBJECT)l: 12 T:7f
MTPD::SendObject: len:340510
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
............................................
Timeout: 3024631
>> 4077542
RESP:2005(RSP:OPERATION_NOT_SUPPORTED)l: 12 T:7fNote: it stopped receiving USB packets a little after 3 seconds.
I reset and tried again with a larger file:
Code:
CMD: 100d(SEND_OBJECT)l: 12 T:7a
MTPD::SendObject: len:3161263
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
....................
Timeout: 3033143
>> 4034826
RESP:2005(RSP:OPERATION_NOT_SUPPORTED)l: 12 T:7aAgain it stopped receiving USB packets a little over 3 seconds. Was sort of curious if the Timeout on the PC might change proportionately depending on size... In this case it does not appear to.
Now back to the other drawing board!
mjs513
Senior Member+
Morning @KurtE
Know what you mean. Last night was playing your send_object_large branch and playing around with buffer sizes. Unfortunately couldn't get it to work with NANDs for large files. Transfers would time out when file was almost completely transfered then would have to reset the T4.x. This was using both SPI and QSPI. For the NOR chips seemed to work ok except for the 25Q16jv one (the one on the audio board). To be honest for me your old send_object branch was working better with the NANDs except for the one PDF file I had.
Know what you mean. Last night was playing your send_object_large branch and playing around with buffer sizes. Unfortunately couldn't get it to work with NANDs for large files. Transfers would time out when file was almost completely transfered then would have to reset the T4.x. This was using both SPI and QSPI. For the NOR chips seemed to work ok except for the 25Q16jv one (the one on the audio board). To be honest for me your old send_object branch was working better with the NANDs except for the one PDF file I had.
KurtE
Senior Member+
The interesting thing is I tried the simple version that crashed after 3 plus seconds to Flash to go instead to SDIO. Again 512 bytes at a time.
And it took over 10 seconds to send the file.... So again it appears to be some form of pacing issue...
Note: the #'s, I was curious, so every 1 second I sent an event back to host:
send_Event(MTP_EVENT_UNREPORTED_STATUS); //
Did not help in the QSPI version. but left it in in this run... So as you mentioned the version of code I had earlier that read at a certain pace appeared to help. I have most of the same code in place now, but maybe pacing is different. So again more experiments...
I was running into another issue when I tried it with RAM drive, but figured out I was running out of space.
The SendObjectInfo code is not checking to see if there is enough storage space available, to error it out if not...
Probably should have it call the File systems totalSize and subtract the usedSize and have it error out with either OBJECT_too_large or Store full...
Code:
CMD: 100c(SEND_OBJECT_INFO)l: 20 T:299 : 1 ffffffff
DATA:100c(SEND_OBJECT_INFO)l: 166 T:299 : 0 3000 3130d7 3000 0
SendObjectInfo: 1 4294967295 20003480: 0 3000 0 3130d7 3000 0 0 0 c00 900 18 0 0 0 0 : DSC03361.JPG
RESP:2001(RSP:OK)l: 24 T:299 : 1 ffffffff 52
CMD: 100d(SEND_OBJECT)l: 12 T:29a
MTPD::SendObject: len:3223767
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
.........................................................#.......
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
#................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
...............#.................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
...........................................#.....................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
.............................................#...................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
.#...............................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
..................................................#..............
................................................................
................................................................
................................................................
.................................................#...............
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
............................................#....................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
........................................................#........
................................................................
........................$>> 10296997
RESP:2001(RSP:OK)l: 12 T:29aNote: the #'s, I was curious, so every 1 second I sent an event back to host:
send_Event(MTP_EVENT_UNREPORTED_STATUS); //
Did not help in the QSPI version. but left it in in this run... So as you mentioned the version of code I had earlier that read at a certain pace appeared to help. I have most of the same code in place now, but maybe pacing is different. So again more experiments...
I was running into another issue when I tried it with RAM drive, but figured out I was running out of space.
The SendObjectInfo code is not checking to see if there is enough storage space available, to error it out if not...
Probably should have it call the File systems totalSize and subtract the usedSize and have it error out with either OBJECT_too_large or Store full...
chipaudette
Well-known member
I've had this new MTP code working for a few weeks now. Such a fantastic upgrade!
I'm curious, though...is there a way that the Teensy knows that the PC is connected via USB? Is there a call to the MTP library or to the underlying Teensy USB infrastructure that I can use to see if anything is connected? I might like to light up an LED or something to give visual indication that the PC is connected.
Sorry if the answer is obvious...I'm super new to this.
Chip
I'm curious, though...is there a way that the Teensy knows that the PC is connected via USB? Is there a call to the MTP library or to the underlying Teensy USB infrastructure that I can use to see if anything is connected? I might like to light up an LED or something to give visual indication that the PC is connected.
Sorry if the answer is obvious...I'm super new to this.
Chip
In MTP I'm using
Code:
extern volatile uint8_t usb_configuration;it is 0 when not initialized
KurtE
Senior Member+
I was wondering if we should build that capability into the mtp object?
That is do like we do in the USB Host cases or the !Serial type cases...
That is maybe add a member to MTPD:
Sort of like at the beginning:
You can probably do it that way, or you could have it look at some state variable you set (or could set) at either configure time or by processing messages.
Example you could probably trigger off of if we have opened a session: bool could be something like return sessionID_
Currently local variable in the .cpp file should probably be moved to be static member of the MTPD_ class...
That is do like we do in the USB Host cases or the !Serial type cases...
That is maybe add a member to MTPD:
Sort of like at the beginning:
Code:
public:
explicit MTPD(MTPStorageInterface* storage): storage_(storage) {}
[COLOR="#FF0000"] operator bool() { return usb_configuration;}[/COLOR]You can probably do it that way, or you could have it look at some state variable you set (or could set) at either configure time or by processing messages.
Example you could probably trigger off of if we have opened a session: bool could be something like return sessionID_
Currently local variable in the .cpp file should probably be moved to be static member of the MTPD_ class...
This is partially done in Serial, but I agree that checking for open session is a good idea, especially as PC will time out MTP when session is not established for some time (have not established timeout value, but encountered PC timeout few times)
KurtE
Senior Member+
@WMXZ (and others) - I hacked up a bool like I mentioned, which I have not pushed up yet in my WIP branch. I May push it also into the branch that I have an OpenPR on.
Still playing some with different techniques for sendObject to make it work reasonably with slower file systems and not having the PC timeout the whole connection.
I did find that I could get it to work when I paced the reads to about 4-5ms per read max. If I went to 7.5ms it would fail. Also there appears to be a max time between when I get the last read and I respond to it saying that the file was received OK. Again I have played with code that would return after it received everything and then continue to write out the rest of the buffers in the MTPD::loop code.
Which when I transferred a big > 3MB file, might be 30 seconds...
Some of the problems of that approach include:
a) No current indication that there is stuff still being saved away.
b) I allocated extmem_malloc a buffer large enough for whole file. May not have external memory and/or not enough available to allocate that much memory.
c) The file is not closed until it is fully output. Only when closed does the MTPD/storage code update the Record data with the real size, which started off as 0 when allocated. I do have the current code send an event to host saying that parameter changed, which worked. as if you hit F5 it will then get the correct size.
...
Performance of the sendObject code? Wondering if the code that currently combines the data received from each of the USB packets into a buffer of 8KB (or 2k or 4K depending on how configured), speeds up the sendObject? That is, the current released code when it sees there is a USB read transfer that is ready, it calls the function, which memcpy the data into your own 512 byte buffer and then it memcpy it from this buffer to the appropriate place in the 8KB, and since the first 512 byte read also included header data of 12 bytes, there is code to know that you need to know that a read will fill in the last 12 bytes of your 8KB and only memcpy that first part in, and then after that transfer copy the remaining 512-12 byte back to the start of your write buffer... (Note: this is the T4.x version)
Note: my version of this above tries to bypass some of these memory copy functions, by passing in the right spot within the larger buffer to receive the data.
So was wondering if you noticed any speed differences versus the T3.x version, which simply called the write method first with the 500 bytes remaining of the first packet and then with the 512 byte full packets of the next ones, until the last one? Again I am assuming that this comparison is mainly dealing with SD cards.
With LittleFS I am wondering if there is any differences? I probably need to study code again to see if there is anywhere in the code that some fixed size writes that are aligned to some alignment somehow bypasses overhead within the FS?
So I am almost thinking of doing a hybrid implementation. maybe when I call addFileSystem, I can pass in an optional parameter that maybe says this is a slower FS and to use some of the idle read ahead and pacing code, for faster FS, probably stick with current stuff, maybe with a few twists?
Thoughts?
First coffee and ...
Still playing some with different techniques for sendObject to make it work reasonably with slower file systems and not having the PC timeout the whole connection.
I did find that I could get it to work when I paced the reads to about 4-5ms per read max. If I went to 7.5ms it would fail. Also there appears to be a max time between when I get the last read and I respond to it saying that the file was received OK. Again I have played with code that would return after it received everything and then continue to write out the rest of the buffers in the MTPD::loop code.
Which when I transferred a big > 3MB file, might be 30 seconds...
Some of the problems of that approach include:
a) No current indication that there is stuff still being saved away.
b) I allocated extmem_malloc a buffer large enough for whole file. May not have external memory and/or not enough available to allocate that much memory.
c) The file is not closed until it is fully output. Only when closed does the MTPD/storage code update the Record data with the real size, which started off as 0 when allocated. I do have the current code send an event to host saying that parameter changed, which worked. as if you hit F5 it will then get the correct size.
...
Performance of the sendObject code? Wondering if the code that currently combines the data received from each of the USB packets into a buffer of 8KB (or 2k or 4K depending on how configured), speeds up the sendObject? That is, the current released code when it sees there is a USB read transfer that is ready, it calls the function, which memcpy the data into your own 512 byte buffer and then it memcpy it from this buffer to the appropriate place in the 8KB, and since the first 512 byte read also included header data of 12 bytes, there is code to know that you need to know that a read will fill in the last 12 bytes of your 8KB and only memcpy that first part in, and then after that transfer copy the remaining 512-12 byte back to the start of your write buffer... (Note: this is the T4.x version)
Note: my version of this above tries to bypass some of these memory copy functions, by passing in the right spot within the larger buffer to receive the data.
So was wondering if you noticed any speed differences versus the T3.x version, which simply called the write method first with the 500 bytes remaining of the first packet and then with the 512 byte full packets of the next ones, until the last one? Again I am assuming that this comparison is mainly dealing with SD cards.
With LittleFS I am wondering if there is any differences? I probably need to study code again to see if there is anywhere in the code that some fixed size writes that are aligned to some alignment somehow bypasses overhead within the FS?
So I am almost thinking of doing a hybrid implementation. maybe when I call addFileSystem, I can pass in an optional parameter that maybe says this is a slower FS and to use some of the idle read ahead and pacing code, for faster FS, probably stick with current stuff, maybe with a few twists?
Thoughts?
First coffee and ...
mjs513
Senior Member+
@KurtE
Have no ideas beyond what I documented with testing on flash chips. It perplexing. Not sure you are ever going to get a 100% solution. From testing with your previous version without buffers it would only get stuck periodically with certain types of files. Not sure what else I can contribute - this is giving me a headache now.
Have no ideas beyond what I documented with testing on flash chips. It perplexing. Not sure you are ever going to get a 100% solution. From testing with your previous version without buffers it would only get stuck periodically with certain types of files. Not sure what else I can contribute - this is giving me a headache now.
defragster
Senior Member+
I can see this being a headache. SD cards have known block sizes for efficient write - Little FS is different in that small files fit in the DIR header and larger allocate blocks based on the 'media' involved block size - but those blocks are shared with a 'header' to ID the block - then the data fills 'the rest'.
What constitutes a small file depends on the block size and other 'logic' in the LFS code. These are large files so the best write size would be 'data stored per block' - so that a block isn't commited then needs re-written when it grows to the next block.
Haven't read the LFS docs enough to know the size of the header for 'data' blocks - but with alternate sizes of 'media' blocks it would vary even so.
What constitutes a small file depends on the block size and other 'logic' in the LFS code. These are large files so the best write size would be 'data stored per block' - so that a block isn't commited then needs re-written when it grows to the next block.
Haven't read the LFS docs enough to know the size of the header for 'data' blocks - but with alternate sizes of 'media' blocks it would vary even so.
KurtE
Senior Member+
@mjs513 @defragster @WMXZ @PaulStoffregen ...
Mike I think you are right we may never get a 100% in all cases.
The question is can we get closer.
As I mentioned, I am thinking of a Hybrid version, where maybe we control how things are done by store...
For SDFat like ones probably just run with the current code. Maybe a slight change to avoid one extra memory copy, but again maybe not that important.
For LittleFs like setups, probably go with something like the version I had in other branch and have here, that paces the reads, maybe it computes how long it is taking on average to do writes and tries to pace it at that speed with a maximum wait of something like 4 or 5ms max between reads. Then see if can setup buffering to hold as much of the file as we can in memory.... If the buffer requirements exceed this, then maybe have the code send an event to cancel the transfer. As to hopefully avoid a total error out of the device.
The question to self (and others) is, can I simplify this part to smaller blocks? For example just write out the 500 bytes of the first USB packet and then deal with while multiples of 512 byte USB packets? But again I maybe don't understand enough of under the covers of littleFS, if it would behave differently if for example I have a 50K file if I
first wrote 500 bytes, followed by maybe 24 groups of 2KB followed by what ever was remaining, versus maybe writing the 500 bytes followed by 96 writes of 512 and the remaining bytes like 348,
Or again should I make sure all writes are the same magic size. But again not sure if the overhead per block is the same?
I suppose could run some simple tests to see?
Mike I think you are right we may never get a 100% in all cases.
The question is can we get closer.
As I mentioned, I am thinking of a Hybrid version, where maybe we control how things are done by store...
For SDFat like ones probably just run with the current code. Maybe a slight change to avoid one extra memory copy, but again maybe not that important.
For LittleFs like setups, probably go with something like the version I had in other branch and have here, that paces the reads, maybe it computes how long it is taking on average to do writes and tries to pace it at that speed with a maximum wait of something like 4 or 5ms max between reads. Then see if can setup buffering to hold as much of the file as we can in memory.... If the buffer requirements exceed this, then maybe have the code send an event to cancel the transfer. As to hopefully avoid a total error out of the device.
The question to self (and others) is, can I simplify this part to smaller blocks? For example just write out the 500 bytes of the first USB packet and then deal with while multiples of 512 byte USB packets? But again I maybe don't understand enough of under the covers of littleFS, if it would behave differently if for example I have a 50K file if I
first wrote 500 bytes, followed by maybe 24 groups of 2KB followed by what ever was remaining, versus maybe writing the 500 bytes followed by 96 writes of 512 and the remaining bytes like 348,
Or again should I make sure all writes are the same magic size. But again not sure if the overhead per block is the same?
I suppose could run some simple tests to see?
KurtE
Senior Member+
Quick update:
I still have not finished the above of a hybrid version, will work on that more tomorrow.
I did push up my WIP stuff up in the MEM_send_object_large branch, with a few random things some of which not sure of best way to really implement:
Experiments with:
a) a boolean operator to know if we are connected and have an active session - Note I have not tried this one out yet.
b) sendObjectInfo, add 3rd parameter by reference for returning the new object id, and have this function return status code. It checks to see if there is enough free space to hold the file. Note: this does not compute at all if the actual file will take more space than will actually fit... now doing a copy of a file that is too big to fit will beep error out.
c) Quick and dirty Format enable. Right now I allow the code to register a callback with function if the user chooses the format menu item on a storage. Added callback in example and if they do choose format it will figure out which drive and currenly only with LittleFS will do a quick format. It then uses sldgehammer to call storage_->ResetIndex() to clear out the index... Could/Should maybe instead simply remofe all of the ojbects that are on the storage that was formatted. I don't know if there is some way to have additional format options or not. But will play some more.
Currently real hack, I added an ability to have a cabllback function called if we receive a formatStorage message, which I then call the callback. I then grab what FS the storage is, and then call the quick format.
I still have not finished the above of a hybrid version, will work on that more tomorrow.
I did push up my WIP stuff up in the MEM_send_object_large branch, with a few random things some of which not sure of best way to really implement:
Experiments with:
a) a boolean operator to know if we are connected and have an active session - Note I have not tried this one out yet.
b) sendObjectInfo, add 3rd parameter by reference for returning the new object id, and have this function return status code. It checks to see if there is enough free space to hold the file. Note: this does not compute at all if the actual file will take more space than will actually fit... now doing a copy of a file that is too big to fit will beep error out.
c) Quick and dirty Format enable. Right now I allow the code to register a callback with function if the user chooses the format menu item on a storage. Added callback in example and if they do choose format it will figure out which drive and currenly only with LittleFS will do a quick format. It then uses sldgehammer to call storage_->ResetIndex() to clear out the index... Could/Should maybe instead simply remofe all of the ojbects that are on the storage that was formatted. I don't know if there is some way to have additional format options or not. But will play some more.
Currently real hack, I added an ability to have a cabllback function called if we receive a formatStorage message, which I then call the callback. I then grab what FS the storage is, and then call the quick format.
KurtE
Senior Member+
Another quick update:
I pushed up my current changes to my MEM_send_object_large branch
I have the hybrid version at least partially working, need to test more and maybe cleanup some, and probably some discussions on functionality and how to package...
I currently have the test sketch see if T4.1 and external PSRAM and if so tries to allocate a MB extra buffer. if not tries to allocate a 128kb buffer...
Working with the QSPI drive on this code, the code now uses both the original buffer and extra buffer to read in while it is writing. after it completes receiving the file from host, it will continue to write for up to 2 seconds before it returns from the code and then continues to do writes in the main loop until it completes (1 write per call to loop) Probably need more protection to not start another send object while this one is still active.
Still have the format call back and I can right click on storage like QSPI and choose format, and it does the format... Still need to look at SD...
I pushed up my current changes to my MEM_send_object_large branch
I have the hybrid version at least partially working, need to test more and maybe cleanup some, and probably some discussions on functionality and how to package...
I currently have the test sketch see if T4.1 and external PSRAM and if so tries to allocate a MB extra buffer. if not tries to allocate a 128kb buffer...
Working with the QSPI drive on this code, the code now uses both the original buffer and extra buffer to read in while it is writing. after it completes receiving the file from host, it will continue to write for up to 2 seconds before it returns from the code and then continues to do writes in the main loop until it completes (1 write per call to loop) Probably need more protection to not start another send object while this one is still active.
Still have the format call back and I can right click on storage like QSPI and choose format, and it does the format... Still need to look at SD...
mjs513
Senior Member+
@KurtE
Was doing Google Searches for MTP timeouts and came across couple of libraries that actually have timeout defined. But it looks like the timeouts are on the USB side. Reference:
1. https://github.com/libmtp/libmtp/bl...f1b8f50b3da3afc47/src/libopenusb1-glue.c#L423
2. https://github.com/libmtp/libmtp/search?q=timeout
3. https://github.com/libmtp/libmtp/search?q=timeout
Those references are the ones I like the best since it was reminiscent of what we do. Not sure if this helps any.
EDIT: Ok may have spoken too soon. Looks like MTP and usb1 have timeouts defined at 60ms:
Maybe changing the timeouts to be longer?
Was doing Google Searches for MTP timeouts and came across couple of libraries that actually have timeout defined. But it looks like the timeouts are on the USB side. Reference:
1. https://github.com/libmtp/libmtp/bl...f1b8f50b3da3afc47/src/libopenusb1-glue.c#L423
2. https://github.com/libmtp/libmtp/search?q=timeout
3. https://github.com/libmtp/libmtp/search?q=timeout
Those references are the ones I like the best since it was reminiscent of what we do. Not sure if this helps any.
EDIT: Ok may have spoken too soon. Looks like MTP and usb1 have timeouts defined at 60ms:
Code:
int MTPD::pull_packet(uint8_t *data_buffer)
{
while(!usb_mtp_available());
return usb_mtp_recv(data_buffer,60);
}
int MTPD::fetch_packet(uint8_t *data_buffer)
{
return usb_mtp_recv(data_buffer,60);mjs513
Senior Member+
@KurtE
Just did some testing with your new and improved mem_send_obj branch:
1. QSPI: 25Q64, 25Q128, M02 (NAND) - was able to send PDFs up 4.9MB without a problem. All copied over without a problem.
2. QSPI: 25Q512 - could copy over any large files without it crashing/restarting. Was able to copy a 214kb.
Haven't tested with SPI yet though.
Just did some testing with your new and improved mem_send_obj branch:
1. QSPI: 25Q64, 25Q128, M02 (NAND) - was able to send PDFs up 4.9MB without a problem. All copied over without a problem.
2. QSPI: 25Q512 - could copy over any large files without it crashing/restarting. Was able to copy a 214kb.
Haven't tested with SPI yet though.
KurtE
Senior Member+
Morning @mjs513 - Which configuration? T4.1? With our without PSRAM?
The reason I am asking is the test sketch there looks for PSRAM and tries to allocate a larger buffer(extmem_malloc). If it does not have that memory it tries to malloc a smaller buffer.
Note: My changes are so far only in the IMXRT part of the code.
The T3.x part of the code still uses the simpler read a USB packet write to MTP approach.
As for Timeouts: The code that reads on yield, has its own way of doing timeouts:
That is the code does not call pull_packet, but instead does it's own call to: if (usb_mtp_available()) {
and only then does it call usb_mtp_recv, but I do still pass in 60. Figured it probably did not matter as I already checked that it has a packet ready for me. with the available call.
The main part of the send using yield: bool MTPD::SendObjectWithYield()
Has another timeout, that if it sits in the loop still waiting for data for something like 10 seconds without doing anything, it bails.
There is also some other internal timeouts that I put in that are sort of strange, currently hard coded. That is:
Suppose that you transfer a file that is example 1MB in size and we have the code to pace doing the reads. Now suppose we receive the last data for the file and we still have maybe 256KB (maybe more left to write out). How long can I now continue to do writes to the storage, before returning a response to the host before the host gets mad. It would be nice to have the whole file output, as then when the host gets our response, it will ask for the size and currently with file open we give them size 0...
So the code now has a hard coded 2 seconds. That I will continue to do writes in the main Send Object function for 2 seconds before returning. Hopefully in most cases that is enough time to complete. But if not, the main loop code will check to see if it has an active transfer, if so it will start the next write... When in this case it writes out the last buffer, it closes the file and sends an event to the host saying that the size parameter changed. Wish windows updated it directly but if you hit F5 it shows the final size...
There are still things to cleanup, like what happens if the host tries to start up a new transfer while I am still processing the last one...
The reason I am asking is the test sketch there looks for PSRAM and tries to allocate a larger buffer(extmem_malloc). If it does not have that memory it tries to malloc a smaller buffer.
Note: My changes are so far only in the IMXRT part of the code.
The T3.x part of the code still uses the simpler read a USB packet write to MTP approach.
As for Timeouts: The code that reads on yield, has its own way of doing timeouts:
That is the code does not call pull_packet, but instead does it's own call to: if (usb_mtp_available()) {
and only then does it call usb_mtp_recv, but I do still pass in 60. Figured it probably did not matter as I already checked that it has a packet ready for me. with the available call.
The main part of the send using yield: bool MTPD::SendObjectWithYield()
Has another timeout, that if it sits in the loop still waiting for data for something like 10 seconds without doing anything, it bails.
There is also some other internal timeouts that I put in that are sort of strange, currently hard coded. That is:
Suppose that you transfer a file that is example 1MB in size and we have the code to pace doing the reads. Now suppose we receive the last data for the file and we still have maybe 256KB (maybe more left to write out). How long can I now continue to do writes to the storage, before returning a response to the host before the host gets mad. It would be nice to have the whole file output, as then when the host gets our response, it will ask for the size and currently with file open we give them size 0...
So the code now has a hard coded 2 seconds. That I will continue to do writes in the main Send Object function for 2 seconds before returning. Hopefully in most cases that is enough time to complete. But if not, the main loop code will check to see if it has an active transfer, if so it will start the next write... When in this case it writes out the last buffer, it closes the file and sends an event to the host saying that the size parameter changed. Wish windows updated it directly but if you hit F5 it shows the final size...
There are still things to cleanup, like what happens if the host tries to start up a new transfer while I am still processing the last one...
mjs513
Senior Member+
Morning @KurtE.
QSPI testing was done with a T4.1 without any PSRAM. Unfortunately I didn't add the PSRAM chip when I did that setup for testing QSPI. But even without it did well on transferring megabyte files except for the 25Q512 Flash. Not sure why that one is giving me a problem. Will retest with a T4.1 with the PSRAM, at least for the 25Q512 chip and probably the 25Q64.
Initial SPI testing without the PSRAM on a T4.0, basically failed on a 4.8MB file for the same FLASH chips. Will retest that as well with PSRAM. But a little concerned that if we need PSRAM to get it to transfer large files its going to be limited to the T4.1. Know this is just testing but had to say it.
Stay tuned.
PS - had to take puppy to Vet for shots so just getting back home, so back to it.
QSPI testing was done with a T4.1 without any PSRAM. Unfortunately I didn't add the PSRAM chip when I did that setup for testing QSPI. But even without it did well on transferring megabyte files except for the 25Q512 Flash. Not sure why that one is giving me a problem. Will retest with a T4.1 with the PSRAM, at least for the 25Q512 chip and probably the 25Q64.
Initial SPI testing without the PSRAM on a T4.0, basically failed on a 4.8MB file for the same FLASH chips. Will retest that as well with PSRAM. But a little concerned that if we need PSRAM to get it to transfer large files its going to be limited to the T4.1. Know this is just testing but had to say it.
Stay tuned.
PS - had to take puppy to Vet for shots so just getting back home, so back to it.
KurtE
Senior Member+
Maybe I need to get back to testing as well.
How did these fail? Did Windows completely drop you? With the disconnect? Or did I drop out or???
One thing that confused me at one point was if you try to copy a file to the storage that is > free space, than I now return a out of space error on the first message, which when I do it on windows it beeps with the error (does not show error message or like).
Another thing that I know we are probably not handling totally correct, is suppose you try to copy a 2MB file and the FS says it has lets say 2.2MB Free (total - used) but then suppose it turns out LittleFS will take 3MB to actually store the file. Do we properly detect this. Does it need to continue to read in the complete file and then error out...
I wish there was some spec or site that sort of specification on how long devices have for timeouts and the like. Will play some more.
Are you mainly testing with the 4 chip SPI memory board? I have been mostly doing with the slow QSPI board I have.
How did these fail? Did Windows completely drop you? With the disconnect? Or did I drop out or???
One thing that confused me at one point was if you try to copy a file to the storage that is > free space, than I now return a out of space error on the first message, which when I do it on windows it beeps with the error (does not show error message or like).
Another thing that I know we are probably not handling totally correct, is suppose you try to copy a 2MB file and the FS says it has lets say 2.2MB Free (total - used) but then suppose it turns out LittleFS will take 3MB to actually store the file. Do we properly detect this. Does it need to continue to read in the complete file and then error out...
I wish there was some spec or site that sort of specification on how long devices have for timeouts and the like. Will play some more.
Are you mainly testing with the 4 chip SPI memory board? I have been mostly doing with the slow QSPI board I have.
mjs513
Senior Member+
For SPI testing of the 2G, 1G and 512MBit chips I used the breakout board with the 4 chips on it. For the other chips like the 25Q64, 128 I used a breadboard. Used a T4.1 on both.
For QSPI testing I used my T4.1 breakout with wires from the QSPI pads for FLASH to headers. I soldered wirewrap wires from a T4.1 to the headers. I used one of those Adafruit permaboards for that one.
When it failed to transfer it either died and could do anything else until I repowered the T4.1. In some other cases it would restart by itself. Think the auto restart was with SPI. In one or two cases I saw on the serial monitor unknown command before it hung.
I did make sure there was nothing else on the chips before I did the transfers so wouldn't run into the issue with out of memory.
For QSPI testing I used my T4.1 breakout with wires from the QSPI pads for FLASH to headers. I soldered wirewrap wires from a T4.1 to the headers. I used one of those Adafruit permaboards for that one.
When it failed to transfer it either died and could do anything else until I repowered the T4.1. In some other cases it would restart by itself. Think the auto restart was with SPI. In one or two cases I saw on the serial monitor unknown command before it hung.
I did make sure there was nothing else on the chips before I did the transfers so wouldn't run into the issue with out of memory.
KurtE
Senior Member+
I am running into a bunch of screwy errors with the SPI version as well.
I may need to see what I am running as I am failing on smaller files as well.
I am thinking there is probably a bug
in my code, maybe I stopped the prefetch and did not restart. Will take a look.
Will also play some with the prefetch timing code. Right now I have it as one hard coded value...
May change to do something like: I am writing 2KB at at time, how long did the last N writes take, and each read is 512b. so what in this case have the delays maybe computed, like 1/4 of time it takes to do a write with MAX of 5ms... and see how that works. I was testing before on really slow stuff, so probably simple issue of not so slow showing an issue...
I may need to see what I am running as I am failing on smaller files as well.
I am thinking there is probably a bug
in my code, maybe I stopped the prefetch and did not restart. Will take a look. Will also play some with the prefetch timing code. Right now I have it as one hard coded value...
May change to do something like: I am writing 2KB at at time, how long did the last N writes take, and each read is 512b. so what in this case have the delays maybe computed, like 1/4 of time it takes to do a write with MAX of 5ms... and see how that works. I was testing before on really slow stuff, so probably simple issue of not so slow showing an issue...
defragster
Senior Member+
Is this using MTP as included in TD 1.54 b6 - or has it morphed or using another code base?
Have the p#645 "2. 25Q64 and 25Q512 failed to transfer the same file." Flash chips been formatted? ... myfs.formatUnused(0) : before the test started?
During FLASH work for LFS during wait()'s yield per seconds in LFSIntegrity is showing counts of :: yps=1259595
Is there anything that could be done or added in a local void yield() on occasion to send the MTP Host some warm fuzzy feeling? Perhaps reset a timer after any message to Host that makes it feel warm and fuzzy - and in the yield() code if that timer is bigger than 'x' - reset timer and send some event that might keep serve as a 'Keep Alive'?
Also - since there are stalls/hangs - perhaps FrankB's HardFault code should be added for debug to make sure it isn't a Fault and not just a "maybe I stopped ..." code failure?
I'll look at some stuff if I can get help to SYNC on exactly what is under test now ... github link to sources and update install steps?
Have the p#645 "2. 25Q64 and 25Q512 failed to transfer the same file." Flash chips been formatted? ... myfs.formatUnused(0) : before the test started?
During FLASH work for LFS during wait()'s yield per seconds in LFSIntegrity is showing counts of :: yps=1259595
Is there anything that could be done or added in a local void yield() on occasion to send the MTP Host some warm fuzzy feeling? Perhaps reset a timer after any message to Host that makes it feel warm and fuzzy - and in the yield() code if that timer is bigger than 'x' - reset timer and send some event that might keep serve as a 'Keep Alive'?
Also - since there are stalls/hangs - perhaps FrankB's HardFault code should be added for debug to make sure it isn't a Fault and not just a "maybe I stopped ..." code failure?
I'll look at some stuff if I can get help to SYNC on exactly what is under test now ... github link to sources and update install steps?
mjs513
Senior Member+
@defragster
Currently testing with @KurtE's branch: https://github.com/KurtE/MTP_t4/tree/MEM_send_object_large. Basically, the MTP version you have in your libraries folder and replace with @KurtE's. The MTP library was not installed as part of Teensyduino. The changes to the core files necessary to make it work were though so you don't have to worry about that part. Thang god for Beta6.
All the chips were previously formatted so should not be a problem but easy enough to check. EDIT: reformated but still fails.
Paul didn't pull in Frank's "Print Hardfault" PR so would have to do it manually to get it to work.
Currently testing with @KurtE's branch: https://github.com/KurtE/MTP_t4/tree/MEM_send_object_large. Basically, the MTP version you have in your libraries folder and replace with @KurtE's. The MTP library was not installed as part of Teensyduino. The changes to the core files necessary to make it work were though so you don't have to worry about that part. Thang god for Beta6.
All the chips were previously formatted so should not be a problem but easy enough to check. EDIT: reformated but still fails.
Paul didn't pull in Frank's "Print Hardfault" PR so would have to do it manually to get it to work.