Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 20 of 20

Thread: Fast Data Logging over USB

  1. #1
    Junior Member
    Join Date
    Dec 2018
    Location
    PNW
    Posts
    7

    Fast Data Logging over USB

    I've recently started playing with the Teensy 3.6 and Arduino IDE configured as a 'USB Serial' device at 180MHz, optimized 'Faster'. I am trying to put together a Time of Flight acoustic measuring system which needs quick and well timed analog measurements. So, naturally I've been using Pedvide's ADC library with PDB. So far so good, just measuring the analog write pin to test. Teensy seems to have no trouble keeping up with many hundreds of KSPS. My problem comes from trying to log the data. I'm trying to send the data over USB to my laptop so I can start to run the analysis. I have set up the ADC library to 10 bit measurements, and unless I throttle the sampling frequency to 10 Ksps I cannot get all the data to my computer. I figure that with 2 bytes for data and 1 byte for delimiting per sample, I should be able to transfer at least 300 Ksps, but I drop a lot of data across the USB unless the speed is quite slow. Below is the Teensy Code (many thanks to all the excellent forum posts about ADC):

    Code:
    #include <ADC.h>
    #include <RingBuffer.h>
    
    // connect out_pin to adc_pin, PWM output on out_pin will be measured
    // via adc_pin.
    
    const uint8_t adc_pin = A9;
    const uint8_t out_pin = 2;
    
    ADC adc;
    
    
    RingBuffer *buffer = new RingBuffer;
    
    volatile size_t write_pos = 0;
    volatile size_t read_pos = 0;
    
    bool flag = false;
    bool quitFlag = false;
    
    volatile uint32_t Counter = 0;
    uint32_t readCounter = 0;
    volatile uint16_t adc_val = 0;
    
    const uint32_t pdb_trigger_frequency = 300000;
    const uint32_t seconds = 100;
    
    uint32_t numSamples = pdb_trigger_frequency * seconds;
    
    void setup() {
    
        
        
        pinMode(adc_pin, INPUT);
        Serial.begin(9600);
        delay(3000);
        Serial.println("Starting");
    
        adc.setAveraging(1);
        adc.setResolution(10);
    
        adc.setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED);
        adc.setSamplingSpeed(ADC_SAMPLING_SPEED::HIGH_SPEED);
        adc.adc0->analogRead(adc_pin); // performs various ADC setup stuff
    
        analogWriteFrequency(out_pin, 5000);
        analogWrite(out_pin, 128);
        
    
        delay(2000);
        Serial.println(millis());
    
        adc.enableInterrupts(ADC_0);
        //NVIC_SET_PRIORITY(IRQ_ADC0,0);
        //NVIC_SET_PRIORITY(IRQ_PDB,2);
        adc.adc0->resetError();
        adc.adc0->stopPDB();
        adc.adc0->startPDB(pdb_trigger_frequency);
        
    
        
    
    }
    
    void loop() {
    
    
          if(readCounter < Counter){
            adc_val = buffer->read();
            Serial.println(adc_val);
            readCounter++;
          }
      else{
        if (quitFlag == true){
          Serial.println("OVER");
          Serial.end(); //Does nothing
          quitFlag = false;
        }
      }
         
    }
    //void yield(){}
    
    void adc0_isr() {
      if(Counter<numSamples){
    
        buffer->write(adc.adc0->readSingle());
     
       
      }
          else{
             adc.adc0->stopPDB();
             adc.disableInterrupts(ADC_0);
             adc.adc0->readSingle();
             quitFlag = true;
          }
           Counter++;
    }
    
    void pdb_isr(void) {
        PDB0_SC &=~PDB_SC_PDBIF; // clear interrupt
    }

    For completeness, here is the Python Code (3.7) I am using to catch the data, note that I also attempted this with some other serial monitors (TyCommander among them) and this has been consistently the best for me:


    Code:
    import time
    
    flag = 1
    with open('logFile', 'w') as log:
        time.sleep(1)
        # Don't fight the Kernal, wait some seconds for prepare device
    
        with open("/dev/ttyACM0", "r") as readBuff:
            while flag:
                ans = readBuff.readline()
                if ans:
                    if ans[:-1] == "OVER":
                            flag = 0
                    else:
                        log.write(ans[:])
            readBuff.close()
        log.close()


    I have also increased the number of NUM_USB_BUFFERS in usb_desc.h to 16 and the number of TX_PACKET_LIMIT in usb_serial.c to 32.

    I am wondering if there is something I can do to get at least 250 Ksps over USB?

    I guess that the USB transfers are colliding with the ADC ISRs. I have tried to run down other possible issues. I have attempted this on both windows and linux, used the USB data transfer tools Paul provided on PJRC website (just north of 1 MB/s as expected), Serial.write instead of serial.println, different numbers of usb buffers and TX packet limits, changed the interrupt priority of ISRs and PDB, I can certainly make things worse, but I can't seem to get better. What is getting me is that the data loss is generally 1-3% across a range of speeds. The data loss gets worse as the sampling speed increases which makes sense, but it doesn't get much worse.

    Here are some results of number of samples transmitted vs received:


    Rx.............Tx...........%
    ---- ............----.........----
    10000........10000......1
    29900........30000......0.997
    49500........50000......0.99
    99000........100000....0.99
    246000......250000....0.984
    295000......300000....0.983
    302000......310000....0.974
    323500......330000....0.980
    340000......350000....0.971

    These are rounded and general averages of a number of runs. Mostly I was running at 300 Ksps, and these percentages are the same whether I sample for 1 second or 100 seconds. The losses are random throughout the run.

    If anyone has any feedback I would appreciate it. Thanks in advance.

  2. #2
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,358
    Not sure how the RingBuffer works?

    The _isr() may be injecting more than one per loop - and loop() is only pulling one each time?

    This cannot be correct - but TyCommander prints more data without "OVER" printing until the end.
    I picked 21 as the limit on ii because that is close to the ideal USB buffer size of 64 bytes - with 3 bytes per sample, but I oversized the AdcBuff array to allow for data loss as more come in during the write. Then a newline is sent - of course the data is not going out 2 byte binary but as dec to ascii so the byte counts are way off.
    Code:
    uint16_t AdcBuff[42];
    volatile uint32_t ii=0;
    void loop() {
    
            if ( ii > 21 ) {
              for ( int jj=0; jj<11; jj++) {
               Serial.print(AdcBuff[jj]);
               Serial.print(',');
              }
              Serial.println();
              ii=0;
            }
    
          if(readCounter < Counter){
            // adc_val = buffer->read();
            //AdcBuff[ii++] = buffer->read();
            readCounter++;
          }
      else{
        if (quitFlag == true){
          Serial.println("OVER");
          //Serial.end(); //Does nothing
          quitFlag = false;
        }
      }
         
    }
    //void yield(){}
    
    void adc0_isr() {
      if(Counter<numSamples){
    
        // buffer->write(adc.adc0->readSingle());
     AdcBuff[ii++] = adc.adc0->readSingle();
       
      }
          else{
             adc.adc0->stopPDB();
             adc.disableInterrupts(ADC_0);
             adc.adc0->readSingle();
             quitFlag = true;
          }
           Counter++;
    }
    So ignore that - it was just a test to see data was being lost and watch the code with no idea what it is doing. This should be closer to what you want including the void yield(){}.
    The change is adding the "while(readCounter < Counter)". I used comma's because it groups the data better on longer lines in TyCommander with less screen scrolling that will slow it down and make it look jumpy.
    It looks fine to the end with a single value printed after the "OVER".
    Code:
    void loop() {
    
    
          if(readCounter < Counter){
          while(readCounter < Counter){
            adc_val = buffer->read();
            Serial.print(adc_val);
            Serial.print(',');
            readCounter++;
          }
          Serial.println();
        }
      else{
        if (quitFlag == true){
          Serial.println("OVER");
          Serial.end(); //Does nothing
          quitFlag = false;
        }
      }
         
    }
    void yield(){}
    <edit>: The 'single value printed' after "OVER" was probably an _isr() race where the last data came in after "if(readCounter < Counter)" so the "OVER" test should probably read::
    if(readCounter < Counter && quitFlag == true)
    Last edited by defragster; 12-18-2018 at 09:59 AM.

  3. #3
    I modified a program for a test and i am able to send 628571 bytes/sec over USB without any data loss. It only uses 17.8% cpu on a Teensy3.2 at 72Mhz. It's a quite big program i used and it has some overhead, it could probably be faster if it was optimized for the purpose.
    I guess Python can be a problem, i made the test with a C++ program on my PC.

    This is Linux code but you can do something similar in python or Windows to locate to problem. It will tell you if there are data loss on the PC side.

    Code:
    void Serial::printUartStats()
    {
    	struct serial_icounter_struct icount;
    
    	if (ioctl(serial_fd, TIOCGICOUNT, &icount) != -1) {
    		printf("frame %u\n", icount.frame );
    		printf("parity %u\n", icount.parity );
    		printf("overrun %u\n", icount.overrun );
    		printf("brk %u\n", icount.brk );
    		printf("buf_overrun %u\n", icount.buf_overrun );
    		printf("tx %u\n", icount.tx );
    		printf("rx %u\n", icount.rx );
    	}
    }
    Last edited by Jacob.Schultz; 12-18-2018 at 05:01 PM.

  4. #4
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,358
    @Jacob - did you try the change from post above? Was that relevant? When I ran the OP code I saw bursts of data between 'OVER' which suggested the data was coming out - but long after the sampling was done.
    Code:
    void loop() {
        if(readCounter < Counter){
          while(readCounter < Counter){

  5. #5
    I done some fast testing and you cannot blame the USB. The Ringbuffer is certainly not efficient the default size is only 8 bytes (in my version) witch is useless.
    I can recommend a look at https://etlcpp.com. Most of it is good and are difficult to make better.

    RingBuffer.h
    #define RING_BUFFER_DEFAULT_BUFFER_SIZE 8

    The example works much better if i set the ring buffer size to 128 bytes. If i DON'T send anything via USB it will still crash at 300.000 ksps and it will complete at 200.000 ksps. If i do the Serial.println the limit is 150.000 ksps. (compiled with -O3)
    I don't knows the specs for the adc, but it look like some issue at 300k samples with the current example.
    It's possible to save some bandwidth if you pack the data (6 bits is unused for every sample) and frame then in bigger blocks instead of samples.
    My previous test did not depend on the ADC library.
    Last edited by Jacob.Schultz; 12-18-2018 at 10:51 PM.

  6. #6
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,358
    You might try this: github.com/tonton81/Circular_Buffer … circular_buffer_example.ino

    Still not sure if you tried the while? Given the interrupt there will be incoming samples during the output.

    Increased ringbuffer size to 128 and made this change to get a fixed copy of Counter into local var in loop() - seems to run with TyComm:
    Code:
    void loop() {
    
      uint32_t ii = Counter;
      if (readCounter < ii) {
        while (readCounter < ii) {
          adc_val = buffer->read();
          Serial.print(adc_val);
          Serial.print(',');
          readCounter++;
        }
        Serial.println();
      }
      else {
        if (readCounter == Counter && quitFlag == true) {
          Serial.println("OVER");
          //Serial.end(); //Does nothing
          quitFlag = false;
        }
      }
    
    }
    void yield() {}
    Final "if (readCounter == Counter && quitFlag == true) {" noted in post #2 was wrong cut and paste - corrected here.

  7. #7
    Junior Member
    Join Date
    Dec 2018
    Location
    PNW
    Posts
    7
    Thanks for the replies. I did forget to state that I did indeed up the ringbuffer size, it was sitting at a ridiculous 32768, so I dropped it back down to 128 (need to remember what I changed when playing).

    defragster, to your original post, I expected that at some point my data sampling would overrun my ring buffer, but I'm not seeing the results I would expect if that were the cause of my data drop-outs.

    I have tried the while() loop instead of what I posted before and never got better results, before I actually found it to lose more data. When I tried the code you posted however, I got about 295K out of the 300K, so right where I am getting with my own code. Yours does have a benefit of showing where "bunching" of samples is occurring (before each new line) which in the few times I ran it seem to be at random points. At my ridiculous ring buffer size It's a bit interesting that the longest strings of data that I get with your code are hundreds of samples long. One I counted was 850 samples. There was a string of these multiple hundreds of samples in a row before the serial.print() caught back up to the ISR. I'm guessing I was choking up the memory allocations a bit.

    At the more manageable 128 ringbuffer size I actually got 297K of the 300K, and the longest string of bunched samples was 30. It does take 6-12 loops before the counter catches back up so something is causing a bit of a backup, may not matter though. I will need to play with it more tomorrow.

    Jacob.Schultz, I will need to spend some more time looking at everything you posted. I'm still new to Linux so getting around the ports is taking me longer than I would like.

  8. #8
    Remember to set your interrupt priorities on the ADC it's important. It makes your first example work without problems. It should be lower than the usb.

    NVIC_SET_PRIORITY(IRQ_ADC0, 30);

  9. #9
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,358
    I'm just viewing on Windows with TyComm - but the last posted code looks more consistent in common group sizes - where the while keys on ii captured before the while is entered. USB buffers will send when full so keeping a full one on hand - then breaking often enough to make sure allowing them to send.

    Wondering what the ringbuffer does - when underflowed? If it overflows data is lost - but yet in the end as many samples are pulled as pushed in.

    It would only take a couple of lines to move to tonton81's CircularBuffer code that may be a bit better behaved ( I hope ) and maybe even faster.

  10. #10
    Junior Member
    Join Date
    Dec 2018
    Location
    PNW
    Posts
    7
    Yeah, definitely need to try the new circular buffer code, and maybe rewrite my logging in C++. I made sure my ADC priority was set to 30 as Jacob.Schultz said, but I'm still stuck at 295K Rx for 300K Tx, but if I try on TyCommander, I only get about 200K, so that's a bit weird as well. It'll probably be a couple days before I can run down if it's my software, my hardware, or something else. I'll come back and post when I can find the time to run this down.

  11. #11
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,358
    Given a read by Python code - not human terminal - Transmitting the data as 2 byte Binary instead of ascii could generally cut down on the transfer volume depending on the mix of values sent - especially if sent in groups with fewer terminator characters.

  12. #12
    Senior Member+ MichaelMeissner's Avatar
    Join Date
    Nov 2012
    Location
    Ayer Massachussetts
    Posts
    2,859
    Quote Originally Posted by defragster View Post
    Given a read by Python code - not human terminal - Transmitting the data as 2 byte Binary instead of ascii could generally cut down on the transfer volume depending on the mix of values sent - especially if sent in groups with fewer terminator characters.
    Though you may need to watch out for certain characters that may/may not be interpreted by the OS (control-s, control-q, newline, tab, etc.). In theory, you should be able to use a RAW protocol and not have to worry about such things, but sometimes they can come back and bite you.

  13. #13
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,358
    Indeed - may need some care and attention for the receiver for both bytes to clear if read as chars not binary (i.e. print() versus write() from Arduino). And still preserve 0xff or something for the sync markers between, but 10 bit analog read values leave room for adjustment.

  14. #14
    The ringbuffer speed is not bad at all, you can't get it much faster. I compared these two and the speed is identical on ARM. On Intel the ADC circular buffer is 10% faster witch surprised me.

    Code:
    /*
     * queue.h
     *
     *  Created on: Nov 3, 2018
     *      Author: jsa
     */
    
    #ifndef QUEUE_H_
    #define QUEUE_H_
    
    #include <defs.h>
    
    template<class T, int max_size>
    class Queue {
    public:
    	Queue() :
    		m_nItems(0),
    		m_head(0),
    		m_tail(0)
    	{}
    	void push(const T& item) {
    		ASSERT_ID(ERR_QUEUE_OVERFLOW, m_nItems < max_size);
    		m_buf[m_tail] = item;
    		if (++m_tail == max_size) {
    			m_tail = 0;
    		}
    		m_nItems++;
    	}
    	void pop() {
    		ASSERT_ID(ERR_QUEUE_UNDERFLOW, empty() != 1);
    		if (++m_head == max_size) {
    			m_head = 0;
    		}
    		m_nItems--;
    	}
    	T& front() {
    		return m_buf[m_head];
    	}
    	T& back() {
    		return m_buf[ (m_head == 0) ? max_size : m_head - 1];
    	}
    	bool empty() const {
    		return m_nItems == 0;
    	}
    	bool full() const {
    		return m_nItems == max_size;
    	}
    	uint8_t capacity() const {
    		return max_size;
    	}
    	uint8_t size() const {
    		return m_nItems;
    	}
    	void clear() {
    		m_nItems = 0;
    		m_head = 0;
    		m_tail = 0;
    	}
    private:
    	T m_buf[max_size];
    	uint8_t m_nItems;
    	uint8_t m_head;
    	uint8_t m_tail;
    };
    
    
    
    #endif /* QUEUE_H_ */

  15. #15
    Junior Member
    Join Date
    Dec 2018
    Location
    PNW
    Posts
    7
    Well, I've played a bit more with my code. Python was definitely part of the problem. I switched to the linux shell command: cat /dev/ttyACM0 > logFile.txt
    immediately I got about 295500/300000 samples! but then I changed something...
    So, now I get about 298-299K/300K samples. Changing back to python dropped me again to 295K/300K, so whatever my Python is doing is wrong.

    I have tried before doing serial writes, but they weren't getting me anywhere with the Python and it was more difficult to keep track of the bytes. I'm trying again with C++ now, but saying my C++ is rusty would be generous. So I have a script that will talk to my computer, but I have to work on parsing it and writing to a file, nothing too difficult. I will update when I complete that part.

    I did some more playing with the clocking of my teensy, and once I tried overclocking it my samples tanked. I even found that I was getting more stable results with my clock at 144 or 168 MHz rather than 180 MHz. At 120MHz it dropped a bit as well. Does that make sense?

  16. #16
    Junior Member
    Join Date
    Dec 2018
    Location
    PNW
    Posts
    7
    I've played with this a little more, still can't pin down my problem. My latest attempt was using Serial.write as opposed to Serial.println and using C++ to capture the data instead of python or cat.


    Code:
    #include <ADC.h>
    #include <RingBuffer.h>
    #include <array>
    
    // connect out_pin to adc_pin, PWM output on out_pin will be measured
    // via adc_pin.
    
    const uint8_t adc_pin = A9;
    const uint8_t out_pin = 2;
    
    ADC adc;
    RingBuffer *buffer = new RingBuffer;
    
    
    uint8_t TxBuffer[64];
    uint8_t buffNum = 0;
    uint8_t idx = 0;
    uint8_t buffRead = 0;
    uint8_t endCount = 0;
    
    const uint8_t stopByte[64] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 9, 9, 9, 9};
    
    bool readyRead = false;
    
    bool endFlag = false;
    bool quitFlag = false;
    
    uint32_t Counter = 0;
    uint32_t readCounter = 0;
    uint16_t adc_val = 0;
    
    const uint32_t pdb_trigger_frequency = 100000;
    const uint32_t seconds = 1;
    
    uint32_t numSamples = pdb_trigger_frequency * seconds;
    
    void setup() {
    
        pinMode(adc_pin, INPUT);
        Serial.begin(9600);
        delay(3000);
        //Serial.println("Starting");
        Serial.flush();
        adc.setAveraging(1);
        adc.setResolution(10);
    
        adc.setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED);
        adc.setSamplingSpeed(ADC_SAMPLING_SPEED::HIGH_SPEED);
        adc.adc0->analogRead(adc_pin); // performs various ADC setup stuff
    
    
        analogWriteFrequency(out_pin, 5000);
        analogWrite(out_pin, 128);
    
        delay(2000);
        //Serial.println(millis());
    
        adc.enableInterrupts(ADC_0);
        NVIC_SET_PRIORITY(IRQ_ADC0,30);
        adc.adc0->resetError();
        adc.adc0->stopPDB();
        adc.adc0->startPDB(pdb_trigger_frequency);
        
    }
    void loop() {
    
    
    
    
    
      while(readCounter < Counter){
        
          if(buffNum < 63){
              adc_val = buffer->read();
              TxBuffer[buffNum] = (0x00ff&adc_val);
              buffNum++;
              TxBuffer[buffNum] = (adc_val>>8);
              buffNum++;
              readCounter++;
          }
    
          
          else{
            Serial.write(TxBuffer,64);
          //  Serial.println();
            buffNum = 0;
          }
    
      }
    
        if (readCounter == Counter && quitFlag == true){
          Serial.flush();
          Serial.write(stopByte,64);
          Serial.end(); //Does nothing
          quitFlag = false;
        }
         
    }
    
    void yield(){}
    
    void adc0_isr() {
      if(Counter<numSamples){
    
          buffer->write(adc.adc0->readSingle());
      }
        
    
          else{
              adc.adc0->stopPDB();
              adc.disableInterrupts(ADC_0);
              adc.adc0->readSingle();
              quitFlag = true;
            
          }
           Counter++;
    }
    
    void pdb_isr(void) {
        PDB0_SC &=~PDB_SC_PDBIF; // clear interrupt
    }
    My C++ code is:

    Code:
    #include <stdio.h>      
    #include <unistd.h>     
    #include <fcntl.h>      
    #include <errno.h>      
    #include <termios.h>    
    #include <iostream>
    #include <fstream>
    #include <string.h>
    #include <chrono>
    #include <linux/serial.h>
    #include <sys/ioctl.h>
    
    using namespace std;
    
    
     int main()
    {
    
        
        int  serial_port = open( "/dev/ttyACM0", O_RDWR| O_NOCTTY );
        fcntl(serial_port, F_SETFL, 0);
        
        FILE * pFile;
        pFile = fopen ("logCpp.txt","w");
    
        //Define the POSIX structure
        struct termios _serial_options;
        struct serial_icounter_struct icount;
    
        //Read the attribute structure
        tcgetattr(serial_port, &_serial_options);
    
        //Set the baud rate of the port  to 9600
        cfsetispeed(&_serial_options, B4000000);
        cfsetospeed(&_serial_options, B4000000);
                _serial_options.c_cflag |= (CLOCAL | CREAD);
    
        //Define other parameters in order to  realize the 8N1 standard
        _serial_options.c_cflag &= ~PARENB;
        _serial_options.c_cflag &= ~CSTOPB;
        _serial_options.c_cflag &= ~CSIZE;
        _serial_options.c_cflag |= CS8;
    
        //Apply the new attributes 
        tcsetattr(serial_port, TCSANOW, &_serial_options);
        
        errno = 0;
        char read_buffer[64];
       // char subBuff[4] = {0,0,0,0};
      //  char subBuff = 0;
      //  int subIdx = 0;
        bool Flag = 1;
        
        
        auto start = std::chrono::high_resolution_clock::now();
        while(Flag==1){
        int rd = read(serial_port , read_buffer, 64);
        
       /* 
            if (ioctl(serial_port, TIOCGICOUNT, &icount) != -1) {
            printf("frame %u\n", icount.frame );
            printf("parity %u\n", icount.parity );
            printf("overrun %u\n", icount.overrun );
            printf("brk %u\n", icount.brk );
            printf("buf_overrun %u\n", icount.buf_overrun );
            printf("tx %u\n", icount.tx );
            printf("rx %u\n", icount.rx );
        }
        */
    
        // Display read data
        if (rd > 0)
        {
           // std::cout << rd << " bytes read: ";
            fprintf(pFile," %d", rd);
            fprintf(pFile," %s", " bytes read: ");
            for (int i = 0; i < rd; i++)
            {
                //if(read_buffer[i] ==13){
                   // subIdx = 0;
                    //int tmp = std::stoi(subBuff);
                   // printf(" %d", tmp);
                  //  printf(" %d", read_buffer[i]);
                 //   std::cout << " ";
                fprintf(pFile,  "%d ", read_buffer[i]);
               // fprintf(pFile," %s", " ");
                
                //}else if(read_buffer[i]!=10){
                   // subBuff[subIdx] = read_buffer[i];
                   // subIdx++;
                //}
            }
            //fprintf(pFile," %s", " # End ");
            fprintf(pFile," %s \n", " # End");
           // std::cout << " # END" << std::endl;
        }
        // Nothing was to be read
        else if (rd == 0)
        {
            std::cout << "No bytes read =(" << std::endl;
        }
        // Couldn't acces the file for read
        else
        {
            if (errno == EAGAIN || errno == EWOULDBLOCK)
            {
                usleep(20); // Sleep for 20 microseconds.
            }
            else
            {
                printf("%d %s \n", errno, strerror(errno));
            }
        }
        auto stop = std::chrono::high_resolution_clock::now();
        auto duration = std::chrono::duration_cast<std::chrono::seconds>(stop - start); 
        if (duration.count() > 10){
            Flag = 0;
        }
        }
    std::cout << std::endl;
    
    }
    Anyway, I tried checking the ioctl as Jacob suggested and never saw any overruns.

    One thing I did realize which I should have caught on to earlier is that Serial.println actually prints out the values as character arrays. With the analog write shifting between 0 and 1024, I was actually printing out '0' and '1' '0' '2' '4', which leads to an average of 2.5 bytes per reading over the course of the file, not 2 like I had been presuming. So, if anything it's remarkable that i can get close to 300K samples using println. It still doesn't explain why I cannot get anything above 10K samples to read without data loss unfortunately.

    I am having trouble with the Serial.write() still. I'm guessing part of it is on my c++ side. If I try and run the Teensy code above and the C++, I get 10-20% of the samples. If I have the arduino serial monitor open while I run the teensy and the c++, I can get above 50%. Still not doing great with it though.

    So, I will probably try a fresh install of my teensy libraries to see if I mucked something up along the way.

  17. #17
    If you are doing 300k samples/sec you are on the limit of available bandwidth. You also have a CR and LF character for every sample. With 5 bytes pr sample (only 3 digits) you are on the limit of a full speed USB. It's more reasonable to use binary transfer.

    I don't understand you problem with data loss, i cannot replicate it. I have tried you program several time at 300k samples without any problems.

    I can recommend you use cfmakeraw() in you terminal (tty) setup or you will have problems with binary data. Try to read man pages or google. execute "man cfmakeraw". if you have them installed.

    Code:
       struct termios tty;
    
        tcgetattr(serial_fd, &tty) < 0) ;
    
        cfsetospeed(&tty, baudrate);
        cfsetispeed(&tty, baudrate);
    
        tty.c_cflag |= (CLOCAL | CREAD);    // ignore modem controls
        tty.c_cflag &= ~CSTOPB;     		// 1 stop bit
        tty.c_cflag &= ~CRTSCTS;    		// no hardware flowcontrol
    
        cfmakeraw(&tty);	// default is no parity, 8-bit characters
    
        tcsetattr(serial_fd, TCSANOW, &tty) != 0);

  18. #18
    Junior Member
    Join Date
    Dec 2018
    Location
    PNW
    Posts
    7
    Jacob.Schultz,

    I have a quick question for you, when you try my code, does your data make sense? I haven't gotten back to my c++ code yet, but I reinstalled arduino and teensyduino to make sure I didn't screw something major up before. I was playing around with the processor speed and while I realize the 3.2 and 3.6 are different, when I slowed down the 3.6 my data stopped making sense. I was still getting about 99% of my data, but the frequency of the measured data was all wrong. With the analog write at 5kHz, I expect about 30 readings of 1023 followed by 30 readings of 0. At lower Mhz, I would get about 11 of 1024 followed by 15 of 0 until I got to about 120k samples. Then it would suddenly shift to 3-4 readings of 1024 and 5-8 of 0 for the remainder. If I tried sampling at 200ksps It would stall at about 150k samples, and at 100ksps It would work as expected (with the 1% data loss). The whole thing would still take 1 sec to run though. So I was just wondering if your 3.2 was measuring appropriately.

    Sample datalog of the stall at 120k samples.

    Code:
    1
    1
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1
    1
    1
    1
    1
    1
    1
    1
    1
    1
    1
    1
    1
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1023
    1
    1
    1
    1
    1
    2
    1
    1
    2
    1
    1
    1
    1
    1
    1
    2
    1
    1023
    1023
    1023
    1
    1
    1
    2
    1
    1023
    1023
    1023
    1
    1
    1
    2
    1
    1023
    1023
    1023
    1
    1
    1
    2
    1
    1023
    1023
    1023
    1
    1

  19. #19
    I found a Teensy 3.6 in my drawer and made a test. The result is the same as before i got exactly 30.000.004 lines every time in the output file, no data loss. I had floating input under the test and the result is between 50-300 most of the time as expected. And no, my output is not changing at different sample rates.

    Starting
    5300
    217
    296
    192
    156
    213
    279
    295
    .
    .
    .
    .
    .
    69
    73
    54
    151
    160
    123
    163
    180
    198
    OVER
    I compiled a exact copy of the code below with following options: "Optimize: fastest" and a 180 Mhz cpu. If you don't get the same result there must be something wrong with you setup on the PC side or a side effect of a hardware problem on you Teensy.

    Code:
    #include <ADC.h>
    #include <RingBuffer.h>
    
    // connect out_pin to adc_pin, PWM output on out_pin will be measured
    // via adc_pin.
    
    const uint8_t adc_pin = A9;
    const uint8_t out_pin = 2;
    
    ADC adc;
    
    
    RingBuffer *buffer = new RingBuffer;
    
    volatile size_t write_pos = 0;
    volatile size_t read_pos = 0;
    
    bool flag = false;
    bool quitFlag = false;
    
    volatile uint32_t Counter = 0;
    uint32_t readCounter = 0;
    volatile uint16_t adc_val = 0;
    
    const uint32_t pdb_trigger_frequency = 300000;
    const uint32_t seconds = 100;
    
    uint32_t numSamples = pdb_trigger_frequency * seconds;
    
    void setup() {
    
        
        
        pinMode(adc_pin, INPUT);
        Serial.begin(9600);
        delay(3000);
        Serial.println("Starting");
    
        adc.setAveraging(1);
        adc.setResolution(10);
    
        adc.setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED);
        adc.setSamplingSpeed(ADC_SAMPLING_SPEED::HIGH_SPEED);
        adc.adc0->analogRead(adc_pin); // performs various ADC setup stuff
    
        analogWriteFrequency(out_pin, 5000);
        analogWrite(out_pin, 128);
        
    
        delay(2000);
        Serial.println(millis());
    
        adc.enableInterrupts(ADC_0);
        NVIC_SET_PRIORITY(IRQ_ADC0,32);
        //NVIC_SET_PRIORITY(IRQ_PDB,2);
        adc.adc0->resetError();
        adc.adc0->stopPDB();
        adc.adc0->startPDB(pdb_trigger_frequency);
        
    
        
    
    }
    
    void loop() {
    
    
          if(readCounter < Counter){
            adc_val = buffer->read();
            Serial.println(adc_val);
            readCounter++;
          }
      else{
        if (quitFlag == true){
          Serial.println("OVER");
          Serial.end(); //Does nothing
          quitFlag = false;
        }
      }
         
    }
    //void yield(){}
    
    void adc0_isr() {
      if(Counter<numSamples){
    
        buffer->write(adc.adc0->readSingle());
     
       
      }
          else{
             adc.adc0->stopPDB();
             adc.disableInterrupts(ADC_0);
             adc.adc0->readSingle();
             quitFlag = true;
          }
           Counter++;
    }
    
    void pdb_isr(void) {
        PDB0_SC &=~PDB_SC_PDBIF; // clear interrupt
    }

  20. #20
    Junior Member
    Join Date
    Dec 2018
    Location
    PNW
    Posts
    7
    cfmakeraw did not fix the issue so I ordered a new 3.6 and a 3.2 to test out. Failed on both the new ones.

    I switched from my laptop to an older win7 desktop that I built a few years ago. I am able to reliably get 300K readings! Except that I still have two issues, probably related:
    It is taking my new code about 7 seconds to transfer the 300K readings (my fault I'm sure, I plan on working on it, won't bother posting the code since it isn't working and not teensy related). And I am still getting the sudden stalled frequency readings. At first I thought that the adc interrupt might be interfering with the analog write, but since I now have plenty of teensy's lying around I used one to just analog write and the other to read and print and got the same behavior. So now I assume that the readings are just too slow, the circular buffer is being overwritten until the 300K samples are reached and the final pattern of the readings are the result. If I can speed up my reading I should be able to validate (since I can't create a buffer large enough to store all readings).

    If anyone has any recommendations on a laptop that won't randomly drop usb data I would appreciate it...Not that the past few weeks haven't been fun.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •