Test of Serial Transfer, fails for large transfers

[luni]

Ok, carry on then.

 

[DrM]

@Luni, we cross-posted We need a mutex.


I have to get started on my actual work now. Thank you for the multi-port example. I will probably implement it in the teensy that is sending the ccd frames, and that will make a big difference in my experiment.

 

[DrM]

@luni Yes, we have something that seems to work. But I am not sure that does not mean there is still a bug.

i think the interesting experiment is to try bidirectional transfers over one or more of those three ports in your example.

 

[Frank B]

Have measured if the transfer fast enough now?

 

[Frank B]

@FrankB no that absolutely does not make the difference. It takes a few extra ops, not significant.

Sorry, no, that is the trick...

But ok.. you don't have to understand how it works, as long it works.
Glad that you have a solution now.

 

[DrM]

@FrankB for the unidirectional port it seems quite fast enough. I will probably measure it, but I can see that it is good. It is much much faster than the same transfers over a single port used bidirectionally. it is a winner.

 

[Frank B]

Perfect

 

[DrM]

@all

So, can anybody tell me how to edit the title of a thread?

If I could, I would change the title to refer to the specific condition under which it fails, "single bidirectional port"

I feel a little uncomfortable to leave it so non-specific

 

[MichaelMeissner]

I think non-superuser users can only edit a post for about an hour or so after it gets posted (you would need to edit the first posting in the thread).

 

[Frank B]

I wouldn't change the title.
There still no bug on Teensy side, wrong is the usage... and we already have thousand of threads "this and that does not work" and in almost every case it turns out to be wrong usage, so we're all used to that.

Don't worry

 

[DrM]

@FrankB Am I right in assuming that you feel the wrong usage is connected to the idea that a hang is somehow cured by re-sending the remaining bytes?

That really is mind-boggling.

We see that once it is hung, none of the sends after that get through. Why would it make any difference whether the next send is the missing bytes or a new packet?


If you want to say it is wrong usage to expect bidirectional data over a usb serial port, well then that might be a philosophical question and we at least can argue.

 

[Frank B]

@FrankB Am I right in assuming that you feel the wrong usage is connected to the idea that a hang is somehow cured by re-sending the remaining bytes?

That really is mind-boggling.

We see that once it is hung, none of the sends after that get through. Why would it make any difference whether the next send is the missing bytes or a new packet?


If you want to say it is wrong usage to expect bidirectional data over a usb serial port, well then that might be a philosophical question and we at least can argue.

No the receiver hung ( dopped data or call it whatever you want), not the Teensy. Teensy continues running, and !tries! to send data in any case. If the receiver can't use them, its not a sender problem.
And.. you changed the user code, not the core.. so thinking there is a problem is ...at least "interesting".

 

[DrM]

@FrankB Well that remains to be proven, actually. I suspect you are right, except that it happens on both Windows and Linux platform and the compiled part of the code is different for those platforms. So, it is going to take some debugging. Also the serial interface when used bidirectionally is extremely slow. So that is also something that needs to be checked out. I would say the useful experiment is to try the bidiectional port test case in c-code and see how that compares to what happens with pyserial.

 

[Frank B]

I don't think that anything remains to be proven. Users tend to think they are the only who use Teensy.. they forget that many thousands use it.
Do you really think you found a problem nobody else found? In 10 years? With Serial, everybody uses?
And, again, as Luni said (read his post again?) - and I say the same: It's not the fact that you're using more "ports" now that makes it work..
The new code justs waits for the PC. That's why I asked "Is it fast enough".

 

[DrM]

@FrankB You can find similar reports on other forums related to Arduino. It turns out it is a common and well known problem.

We really need to do the experiment with c-code and a single port and see if taking pyserial out of the picture fixes it. But also, I wonder how much code we share with arduino in this. I am guessing Paul knows the answer already.

Anyway, I am far too busy right now, I hope I am not drawn in doing it today.

 

[Frank B]

Might be a common problem, It's a common receiver problem.
Other Arduinos use Hardware Serial and an extra Chip converts it to USB. So you're blaming the manufaturers of millions of chips now... really? They are not only in use for arduino - they are everywhere. Millions is too less, i guess... and there are several companies who produce them. But OK, they all have the same bug YOU found... totally clear.

You can see the code in the core. Just take a look?

 

[Frank B]

Might be a common problem, It's a common receiver problem.
Other Arduinos use Hardware Serial and an extra Chip converts it to USB. So you're blaming the manufaturers of millions of chips now... really? They are not only in use for arduino - they are everywhere. Millions is too less, i guess... and there are several companies who produce them. But OK, they all have the same bug YOU found... totally clear.

You can see the code in the core. Just take a look?

EBecause of this new problem you found, serial has optional handshake - software or hardware - from the beginning on. Don't know exactly.. when was it invented? 60yrs ago? 70? And, the software handshake uses one "port" the same as for data.

 

[luni]

I already did further experiments and can confirm that this has nothing to do with one or more serial connections. Checking how much was actually sent is necessary if you don't want to loose any data. Which is obvious since TD will just ignore any further sending 120ms after the PC stops reading. If you simply continue to send your data they go into Nirwana. No need to discuss that further.

BUT: it looks like there is actually something weird going on with buffer contents when TD runs into this timeout. I'm currently doing experiments using a C# receiver to better understand my observations. Will take some time to find out what is happening. @Frank, you mentioned some known Serial bug, do you have more info about that?

 

[luni]

EBecause of this new problem you found, serial has optional handshake - software or hardware - from the beginning on. Don't know exactly.. when was it invented? 60yrs ago? 70? And, the software handshake uses one "port" the same as for data.

@Frank: USB has this handshake built in and should not need an additional layer on top. The host automatically stops reading if its input buffers are full and it stops sending if the input buffers of the receiver are full. (at least this is my understanding). This usually works perfectly and doesn't need much attention. But as mentioned above something does not work as I expect. This can of course be a problem of my expectations . I'll try to dig further into it and report when I know more.

 

[Frank B]

Yes. On the usb transport layer everything is ok. There is layer above that and this is on the Teensy side the user code, and the receiving software on the PC. This layer is the one what faulted. You'll experience the same problem in opposite direction if you read too slow. Edit: Hm, not sure about this anymore.. may be the teensy blocks reception, then and the PCtx buffer grows until it overflows.. depends on the program,what happens, perhaps.

A similar issue is with a pure hardware serial (no usb involved) If you don't read the FIFOs on the serial chip fast enough they overflow. To prevent this, hardware-handshake exists. Software handshake was more for the application level.

Anyway, the application needs to handle this. It just can't send blindly and "hope everthing is ok"

Thats what your program does, it handles the blocking/overflowed usb tx.. Luckily, it seems to be enough.

 

[Frank B]

For fun, a little experiment. I did not want to write code - some could say it may have a bug?

So, I just startet a Teensy with a empty sketch. I want to try the opposite direction.
And yes, teensy just does nothing! It blocks incomming data (or whatever)


Then I used Tera Term:
I sent the 1e6 Bytes file from yesterday via copy & paste.
This is what happened:

It just sent the data. No errormessage, nothing. WIth local echo enabled, it printed the whole file as if there wasn't any problem.



Then I tried H-Term, with the "send file" option.
H-Term said it send 200Bytes sucessfully. But it stopped completely then, and I had to kill it with the task manager, The "cancel" button was dead.
Despite of the hang, H-Term seems to be a bit smarter as it recognized that something is wrong. Perhaps it uses the windows driver in a different way.
Or at least does some checks which Tera does not.


interesting:
You see the "delay" option? What might it be for?

So.. It's an aplication-thing to handle it. Opposite direction, too.

@Frank, you mentioned some known Serial bug, do you have more info about that?

Not really. Some Programs - not all - display garbage if the Teensy sends very very fast. They display PC memory contents between the Teensy Data.
You can try it with the Arduino Serial Monitor. Let it scroll trough fast - you need a fast finger to stop scrolling I've seen java string constants between the correct data, or other things. Like a wrong pointer.
It's not only in the Monitor.. so it can't be a bug there. It must be a Windows problem.

 

[defragster]

@Frank - is this H-Term as in use there? :: https://www.der-hammer.info/pages/terminal.html

The only recurring issue I find in Teensy is trying to trust 'Serial.availableForWrite()' to show availalbe space to buffer to use to throttle sending.

In scanning this I didn't see what code was used to avoid the issue: Just tracking the # bytes sent?

I did get hangs w/repeat sending of 130 byte chunks w/T_4.1 - but that didn't show for Frank. Even with a .flush() when Serial.availableForWrite() said not enough room.

If I get back to that sample I'll try with H-Term.

 

[Frank B]

Can't say much about it..
H-Term was able to handle the T4 speed with sending the 1 million bytes file in 41ms without any waits and "blind" tx.
Yes, the link is correct.

Time to sleep now

 

[Frank B]

@Luni, @Tim - tried your program again:

hey.. the windows bug was showing:

...B<ARM_NONE_EABI-GCC.EXE-WA006570.pf....

OOps.. the whole time I was thinking the bug shows contents of the current thread only... seems to show way more?!? Isn't that a security issue?
Another snippet, as Screenshot:

The Monitor seems to show it more often.. but I've seen this in other programs, too. At the moment, I can't remember, which.. :-(
Edit: It was coolterm. I found the logfile.

 

[defragster]

Installed HTerm - short learning curve - nice features - like RCV CNT. Works okay against T_3.2.

HTerm chokes against the T_4.1 - like IDE and TyComm - after 164040 chars last time ... even added a delayMicroseconds(10); with flush ...

> put the delayMicroseconds(10); on each looped .print and it worked once ... usually once after fresh upload ... 2nd try lost after 1225656


@Frank: not seeing 'other buffer' chars yet. If they are a 'same app' buffer mixup that would be ugly but not a 'security issue' ... hard to tell what the IDE might have generated in its local memory ...