Teensy 4.1 QNethernet requests and SD problems

nieuwemaker

nieuwemaker

Member
A part of my Teensy 4.1 project involves a simple webserver using QNEthernet and SD to physically read files from the SD card. My website is completely served from the Teensy and is located on the SD card. The header of the html looks like this:


<link href="css/app.css" rel="stylesheet"> <!-- GENERIC GLOBAL CSS --> <link href="css/sections/appContainer.css" rel="stylesheet"> <!-- BASE DOCUMENT CSS --> <link href="css/sections/appHeader.css" rel="stylesheet"> <!-- All header markup CSS --> <link href="css/sections/connectTab.css" rel="stylesheet"> <!-- Connect tab CSS --> <link href="css/sections/playTab.css" rel="stylesheet"> <!-- Play tab CSS --> <link href="css/sections/solveTab.css" rel="stylesheet"> <!-- Solve tab CSS --> <!-- JAVASCRIPT --> <script src="http://pixelperfect/js/ipaddress.js"></script> <!-- Controller generated content to prefill data fields --> <script src="js/generic.js"></script> <!-- Generic functions and GLOBALS that are used throughout the APP--> <script src="js/plugins/svg.js"></script> <!-- little SVG library helping me to draw --> <!-- JAVASCRIPT :: model classes --> <script src="js/models/PubSubBase.js"></script> <script src="js/models/Clip.js"></script> <script src="js/models/Channel.js"></script> <script src="js/models/Bank.js"></script> <script src="js/models/Project.js"></script> <script src="js/models/Animation.js"></script> <script src="js/models/Sequencer.js"></script> <!-- JAVASCRIPT :: web components --> <script type="module" src="js/components/toggle-group.js"></script> <script type="module" src="js/components/vertical-slider.js"></script> <script type="module" src="js/components/channel-header.js"></script> <script type="module" src="js/components/clip-view.js"></script> <script type="module" src="js/components/animation-select-modal.js"></script> <script type="module" src="js/components/clip-min-view.js"></script> <script type="module" src="js/components/channel-min-view.js"></script> <script type="module" src="js/components/bank-min-view.js"></script> <script type="module" src="js/components/channel-view.js"></script> <script type="module" src="js/components/bank-select-view.js"></script> <script type="module" src="js/components/bank-view.js"></script> <script type="module" src="js/components/quick-clips-view.js"></script> <script type="module" src="js/components/project-view.js"></script> <script type="module" src="js/components/sequencer-view.js"></script> <script type="module" src="js/components/performance-view.js"></script> <script type="module" src="js/components/status-bar.js"></script> <script type="module" src="js/components/color-select-view.js"></script> <script type="module" src="js/components/default-video-view.js"></script> <!-- JAVASCRIPT :: Manager classes --> <script src="js/models/managers/LocalStorageManager.js"></script> <script src="js/models/managers/AnimationManager.js"></script> <script src="js/models/managers/ControllerManager.js"></script> <script src="js/models/managers/ProjectManager.js"></script> <script src="js/models/managers/MidiManager.js"></script> <script src="js/main.js"></script> <!-- main() function to start APP -->

As the example shows, a lot of files are requested from the SD card and this is where the weird stuff begins. If I only request 1 file it loads perfectly well, but when all these files are requested some files load perfectly, others stop loading at a random amount of lines and even weirder, some code even gets mixed up between files. So CSS code is mixed through JS code for example.

To mitigate the problem I've added a delayMicroseconds(100) after every line that is read and that works! This looks like this:
while (sdFileHandle.available()) { client.write(sdFileHandle.read()); // stream file per byte to http client delayMicroseconds(100); // was experiment to see if files are loaded completely this way ... }

This is complete Bonkers right? The result is that it takes 25 seconds to load the page for 300kb of data, which is unacceptable for my usecase. So what is going on? Teensy cannot multi-task right, so although the browser requests multiple files at the same time, Teensy handles them in a serial order right? And why would that delay even matter? Am I reading too fast from the SD card? Is that even possible using the sdFileHandle.available() that is used in every example?

To provide some more context I will add snippets from the rest of my code so you can see what I do:

Every 5 milliseconds this method is called from the main loop to check if a web-request is being done:
void WebserverController::HandleClient(){ this->client = this->server->available(); if(this->client && this->IsConnected() && this->client.available()){ // step 1: read what the client is requesting // code here to do so // step 2: if it is a request for a weburl this->servePage(getUrl); // --> this handles the web url request this->client.stop(); // saves memory to quit after each request } }

Then it goes to the servePage() method to handle the file that is requested. It uses the standard fileHandler to read the file and send the input directly to the client stream.
void WebserverController::servePage(String url){ if(url == "/"){ url = "/index.html"; } // redirect Root to index String sdName = SD_FILE_PATH + url.substring(1); // get rid of the "/" in front File sdFileHandle = this->sdController->OpenForRead(sdName); if(sdFileHandle){ // The file exists and can be opened String fileType = url.substring(url.indexOf(".")+1); this->sendCorrectHeader(fileType.toLowerCase()); while (sdFileHandle.available()) { client.write(sdFileHandle.read()); // stream file per byte to http client delayMicroseconds(100); // was experim,ent to see if files are loaded completely this way ... } sdFileHandle.close(); client.closeOutput(); } else { // requested source not found. Let's send a friendly message this->sendHTMLHeader(RESPONSE_NOT_FOUND,MIME_HTML); this->client.printf("<html><body>requested HTML document <b>%s</b> does not exist</body></html>",sdName.c_str()); } }

I use the SD.open() function to open a READ handle for the file like this:
File SDController::OpenForRead(String filename){ if(this->initialized){ return SD.open(filename.c_str(),FILE_READ); } return NULL; }

Hopefully someone here understands what is happening here and can help me out. As said. It works, but is very slow and I hate it if things work without knowing why.
 
Side note and pro tip: you can put code inside code blocks with the “</>” button. This makes a “CODE” tag instead of an “ICODE” or font tag, and puts code inside a box instead of making it look like individual and separate (and wrapped) lines.

This also makes it so the lines don’t wrap, making the code much easier to read. I’m currently reading this on my phone and almost all of the lines are wrapped.
 
Last edited:
@shawn thanks for the tip. I was beyond the edit time, so I post it again.

A part of my Teensy 4.1 project involves a simple webserver using QNEthernet and SD to physically read files from the SD card. My website is completely served from the Teensy and is located on the SD card. The header of the html looks like this:
HTML: 
        <link href="css/app.css"  rel="stylesheet">                  <!-- GENERIC GLOBAL CSS -->
        <link href="css/sections/appContainer.css" rel="stylesheet"> <!-- BASE DOCUMENT CSS -->
        <link href="css/sections/appHeader.css" rel="stylesheet">    <!-- All header markup CSS -->
        <link href="css/sections/connectTab.css" rel="stylesheet">   <!-- Connect tab CSS -->
        <link href="css/sections/playTab.css" rel="stylesheet">      <!-- Play tab CSS -->
        <link href="css/sections/solveTab.css" rel="stylesheet">     <!-- Solve tab CSS -->
        <!-- JAVASCRIPT -->
         <script src="http://pixelperfect/js/ipaddress.js"></script>  <!-- Controller generated content to prefill data fields -->
        <script src="js/generic.js"></script>     <!-- Generic functions and GLOBALS that are used throughout the APP-->
        <script src="js/plugins/svg.js"></script> <!-- little SVG library helping me to draw -->
        <!-- JAVASCRIPT :: model classes -->
        <script src="js/models/PubSubBase.js"></script>
        <script src="js/models/Clip.js"></script>
        <script src="js/models/Channel.js"></script>
        <script src="js/models/Bank.js"></script>
        <script src="js/models/Project.js"></script>
        <script src="js/models/Animation.js"></script>
        <script src="js/models/Sequencer.js"></script>
        <!-- JAVASCRIPT :: web components -->
        <script type="module" src="js/components/toggle-group.js"></script>
        <script type="module" src="js/components/vertical-slider.js"></script>
        <script type="module" src="js/components/channel-header.js"></script>
        <script type="module" src="js/components/clip-view.js"></script>
        <script type="module" src="js/components/animation-select-modal.js"></script>
        <script type="module" src="js/components/clip-min-view.js"></script>
        <script type="module" src="js/components/channel-min-view.js"></script>
        <script type="module" src="js/components/bank-min-view.js"></script>
        <script type="module" src="js/components/channel-view.js"></script>
        <script type="module" src="js/components/bank-select-view.js"></script>
        <script type="module" src="js/components/bank-view.js"></script>
        <script type="module" src="js/components/quick-clips-view.js"></script>
        <script type="module" src="js/components/project-view.js"></script>
        <script type="module" src="js/components/sequencer-view.js"></script>
        <script type="module" src="js/components/performance-view.js"></script>
        <script type="module" src="js/components/status-bar.js"></script>
        <script type="module" src="js/components/color-select-view.js"></script>
        <script type="module" src="js/components/default-video-view.js"></script>
        <!-- JAVASCRIPT :: Manager classes -->
        <script src="js/models/managers/LocalStorageManager.js"></script>
        <script src="js/models/managers/AnimationManager.js"></script>
        <script src="js/models/managers/ControllerManager.js"></script>
        <script src="js/models/managers/ProjectManager.js"></script>
        <script src="js/models/managers/MidiManager.js"></script>
        <script src="js/main.js"></script> <!-- main() function to start APP -->

As the example shows, a lot of files are requested from the SD card and this is where the weird stuff begins. If I only request 1 file it loads perfectly well, but when all these files are requested some files load perfectly, others stop loading at a random amount of lines and even weirder, some code even gets mixed up between files. So CSS code is mixed through JS code for example.
To mitigate the problem I've added a delayMicroseconds(100) after every line that is read and that works! This looks like this:
C++: 
while (sdFileHandle.available()) {
        client.write(sdFileHandle.read()); // stream file per byte to http client
        delayMicroseconds(100); // was experiment to see if files are loaded completely this way ...
 }

This is complete Bonkers right? The result is that it takes 25 seconds to load the page for 300kb of data, which is unacceptable for my usecase. So what is going on? Teensy cannot multi-task right, so although the browser requests multiple files at the same time, Teensy handles them in a serial order right? And why would that delay even matter? Am I reading too fast from the SD card? Is that even possible using the sdFileHandle.available() that is used in every example?
To provide some more context I will add snippets from the rest of my code so you can see what I do:
Every 5 milliseconds this method is called from the main loop to check if a web-request is being done:
C++: 
void WebserverController::HandleClient(){
    this->client = this->server->available();
    if(this->client && this->IsConnected() && this->client.available()){
        // step 1: read what the client is requesting
       // code here to do so
        // step 2: if it is a request for a weburl
        this->servePage(getUrl); // --> this handles the web url request
        this->client.stop(); // saves memory to quit after each request
    }
}

Then it goes to the servePage() method to handle the file that is requested. It uses the standard fileHandler to read the file and send the input directly to the client stream.
C++: 
void WebserverController::servePage(String url){
    if(url == "/"){ url = "/index.html"; } // redirect Root to index
        String sdName     = SD_FILE_PATH + url.substring(1); // get rid of the "/" in front
        File sdFileHandle = this->sdController->OpenForRead(sdName);
        if(sdFileHandle){ // The file exists and can be opened
            String fileType = url.substring(url.indexOf(".")+1);
            this->sendCorrectHeader(fileType.toLowerCase());
            while (sdFileHandle.available()) {
                client.write(sdFileHandle.read()); // stream file per byte to http client
                delayMicroseconds(100); // was experim,ent to see if files are loaded completely this way ...
            }
            sdFileHandle.close();
            client.closeOutput();
        } else { // requested source not found. Let's send a friendly message
            this->sendHTMLHeader(RESPONSE_NOT_FOUND,MIME_HTML);
            this->client.printf("<html><body>requested HTML document <b>%s</b> does not exist</body></html>",sdName.c_str());
        }
}

I use the SD.open() function to open a READ handle for the file like this:
C++: 
File SDController::OpenForRead(String filename){
    if(this->initialized){
        return SD.open(filename.c_str(),FILE_READ);
    }
    return NULL;
}

Hopefully someone here understands what is happening here and can help me out. As said. It works, but is very slow and I hate it if things work without knowing why.
 
As an update to this: Copilot suggested I needed to use multiple EthernetClient instances called client (at the moment it is a property of the class) But this did not solve the problem. It suggested I had a concurrency problem, overwriting the EthernetClient and thus mixing up the responses. After re-writing this part it had exactly the same result though ... even when every request was contained in its own EthernetClient. This made me think, that perhaps SD is not returning unique handles to files?
 
Last edited:
That helps, thank you.

I’m going to sound cliché here, but could you please provide a complete and minimal program that demonstrates the problem? I’d like to see your server code.

Question: did you write the server in a similar way to what’s in the QNEthernet examples?
 
I was hoping this was not needed, but creating a minimalistic version could be the only way to debug the problem without having all other components of the software present. I did write it in a similar way as the Examples show with the difference that I use class instances. I also use QNEthernet for REST (GET & POST) / ARTNET and TPM2.NET communication on the same Teensy and this all works perfectly fine.
 
I did a simple test to check if the files are really read in a serial fashion and not in parallel, and I can confirm this. It does not fix my problem yet, but at least one question has been answered.
 
You must log in or register to reply here.
Back
Top