NEWS: I added a touch screen HID device to Teensyduino 1.23 (works also on Linux)

Status
Not open for further replies.

Elmue

Well-known member
After a long discussion with Paul I learned how to write USB HID descriptors and added a touch screen HID device to Teensyduino 1.23.

This solves all the problems of the current Teensyduino 1.23 code.

  • It allows mouse movement with relative and with absolute coordinates at the same time.
  • It works on all operating systems, even on Linux where the X11 server is very buggy and does not accept absolute coordinates from a mouse device.
  • I added additional features.
  • I left the functions of the Mouse class unchanged so you can recompile your sketches without changes.

You have to replace 5 files in the folder ArduinoCompiler/hardware/teensy/avr/cores/teensy3

You find my code in a ZIP file and a detailed article about my work at Codeproject:
http://www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with


Paul:
I would like to see my code in the next version of Teensyduino !
 
Paul,
it is very strange that I did a great work to improve your product and you don't even answer !
 
I suspect Paul has been busy with Arduino release of the day club, and just hasn't had time to surface for air and look at other posts. The last release broke the Teensy serial support, and just as he got one release out, they put out another one.
 
Last edited:
Paul,
it is very strange that I did a great work to improve your product and you don't even answer !

Have a little patience. Like MichaelMeissner said, he's been rather busy with the latest Arduino releases. While he usually works many hours EVERY DAY on answering questions and development work, once in a while he takes some weekend time off. Normally I would just let Paul speak for himself, but once in a while I speak up. As he has said many, many times, "If only there were more hours in the day..."
 
This looks like a nice development - though I don't have call to use it (windows and no touch screen hardware). Your addition is part of what makes the Open Source Teensy so cool - forum users filling a need they see.

I did follow the link and you should look to update the page to use IDE 1.6.5 instead of 1.6.4.
To program the Teensy's processor you need a micro USB cable and you have to install:
1.The Arduino compiler version 1.64. (Download)

Here is the web note on using 1.6.4:
Arduino 1.6.5 support is currently in beta testing.

We do not recommend using Arduino 1.6.4, due to this bug. Arduino 1.6.3 and earlier are not affected, and it's fixed in 1.6.5. Future Teensyduino versions may drop support for 1.6.4.

As noted Paul is busy optimizing, updating and making functional the newly released IDE.
 
> you should look to update the page to use IDE 1.6.5 instead of 1.6.4.

This is not possible.
I modified the code of Teensyduino 1.23.
And when you install it, it tells you that it ONLY supports the versions of the Arduino compiler 1.06, 1.61, 1.63 and 1.64.

Just for that reason I put the direct download links for just these versions of Teensyduino and the Arduino compiler.
 
Because of the IDE bug breaking compiles for so many - everything is moving to 1.6.5 and the new TeensyDuino 1.24 that is the urgency behind the move.
 
Now 3 weeks have passed by.
No answer from Paul.

So I was correct.
PJRC does not want to implement my improvement.

What a pitty !

> Your addition is part of what makes the Open Source Teensy so cool - forum users filling a need they see.

Yes, but PJRC does not care.
 
You are mistaken that PJRC do not care for your submission, you are also mistaken if you think they have a lot of spare time on their hands to rush into submissions etc.

Do you expect Paul to go and collect it from code project website or have you got github and made a pull request? Perhaps you zipped up all requirements (inclusive working example sketch) and emailed it to him?

Pity is spelt with only one t. Patience also has one t. Have some patience for pity's sake!
 
Last edited:
Indeed, additionally as noted the IDE requires updating to 1.6.5, which is why PJRC was forced into the hasty TD update. It was not a planned feature release update. Until the source is shown to be integrated and working of the current tools it doesn't reflect well. Also I'm wondering about the utility for Mac, windows users.

By hosting this post PJRC is posting the code, and with a good writeup it would make a good wiki page, which is a great general high pri task that needs PJRC attention for the sake of all that is Teensy.
 
Hey Elmue, I am also working on a cross-platform touch device. Your zip file only contains the "originals" so I had to use codeproject's "browse code" feature and copy-paste each file, one at a time, then remove the line numbers. Thank goodness Vim can do column-oriented text. I'm just starting my project but plan to test on Windows, OSX and Linux so I'll let you know how it goes.

After it's been tested on all the major platforms it might be worth forking the Teensy repo on GitHub and sending Paul a pull request. Anyway, if it was my project that's how I'd prefer to work with contributions.
 
This is a great addition to the Teensy HID-stack, thank you Elmue. I can see myself using it in the near future.

I'm sure Paul will get around to pulling it in - Many of the pull requests on the audio library from 6 months ago are not included in the stable release yet, it is quite common - I should imagine Paul is waiting for others to test / prove and break, and fix, etc. etc. etc.
 
Has anyone besides Elmue made this work? For me on Linux, Mouse.move() works, but Mouse.moveTo() does nothing, not with the original Teensy files nor with Elmue's modified versions.
 
Hello

It is clear now after several months that Paul was not busy.
He simply is not interested to use my code for touchscreen.
He did not even post one answer in this thread.

That is incredible, because many people need a touchscreen device because relative mouse movement is completely insufficient for most use cases.
It is also incredible because I posted a code that is ready to use and Paul must only merge into the existing code.

I updated my article and my code supports THREE different touchscreens now.
At least one of them should work for you on any operating system.

www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with
 
Awesome progress Elmue - adding more and developing the concept is a promising sign. You've been absent a while with this and one other post today since July.

Looking at your 'codeproject' comment section I see you are finding and resolving issues. TeensyDuino has undergone some updates possibly for HID issues - hopefully you've updated your pull request with all your changes and those needed to stay current with Teensy and Arduino 1.6.7. If you've done that then re-posting that useful info here might have it be found of interest to Paul or others. But dropping bombs and running away won't accomplish much of anything that you might hope for. If you stay active here with your idea and get it tested by a few other folks the ability to include it without ongoing bug and support issues would probably help much more.

The other thread you posted on about a HID problem Paul promptly replied (3rd day after 1/2/16) and that person has not shown any effort in the weeks since. Paul is very responsive to forum user issues - but has his own major projects for the good of all Teensy in the works - and they are not cooperating for his timely completion unfortunately for all.

Re: A new version will come soon member Elmue 13-Feb-16 4:46

Hello
> We are not sure what value to use for the MS REPORTID_MAX_COUNT

This question cannot be answered in short. USB descriptors are highly cryptic and one wrong value makes them fail. To build a correct descriptor and send the correct data you can NOT simply copy a descriptor from Microsoft and paste it into my code. There are much more things to do. Not even adapting the constants is enough. ...

And the descriptor in the MSDN link posted by Georg17 is surely the worst one to start with. There is no need to use such a highly complicated descriptor. This MSDN descriptor is designed for a two finger touchscreen. He surely never used this descriptor successfully because it will not work with my code. I tested the example for the one-finger touch screen for Windows 7 in the MSDN and it does not even work on Windows 7. The MSDN is full of buggy examples!

In the next weeks I will start extending my code to use optionally another descriptor, because I also need that for another project.

So, if you can wait, you can download soon a new version of my code which will allow to use either the current touchscreen device with the usage POINTER or a new one with the usage page DIGITIZERS.

modified 5 days ago.
 
Hello defragster

> hopefully you've updated your pull request with all your changes and those needed to stay current with Teensy

Surely I will NOT make updates for new Teensyduino versions.
I have a lot of other things to do and I don't have time for that.
I posted a code ready for Teensyduino 1.23 at the time when 1.23 was the latest version.



> If you've done that then re-posting that useful info here might have it be found of interest to Paul

It is obvious that he is NOT interested.



> The other thread you posted on about a HID problem Paul promptly replied

Yes, but his reply only said that he will NOT invest time into developing a touchscreen device.



> Looking at your 'codeproject' comment section
> In the next weeks I will start extending my code to use optionally another descriptor

I don't know why you cite an outdated comment.
If you read a little more down you find that this new version is already available.
 
If you want your code to be included follow the formal method of creating a pull request

There's a number of reasons but basically it boils down to Paul being the sole developer. He doesn't have time to pick through your code. GitDiff allows him to see your changes quickly, clearly and how they might effect the Core library.
It also permits others to make useful changes if they are required. A lot of these tend to be around formatting and keeping with the Teensyduino/Arduino style
 
Last edited:
Hello defragster
> hopefully you've updated your pull request with all your changes and those needed to stay current with Teensy

Surely I will NOT make updates for new Teensyduino versions.
I have a lot of other things to do and I don't have time for that.
I posted a code ready for Teensyduino 1.23 at the time when 1.23 was the latest version.


> If you've done that then re-posting that useful info here might have it be found of interest to Paul

It is obvious that he is NOT interested.



> The other thread you posted on about a HID problem Paul promptly replied

Yes, but his reply only said that he will NOT invest time into developing a touchscreen device.



> Looking at your 'codeproject' comment section
> In the next weeks I will start extending my code to use optionally another descriptor

I don't know why you cite an outdated comment.
If you read a little more down you find that this new version is already available.

In this world of the FORUM - where your solution was new and untested with a world of other stuff going on - your personal perception doesn't make anything obvious. You cannot safely infer from Paul's lack of immediate action ...

I worked on a cool interrupt adaptation for something Paul wrote himself - it took time for it to get into the build. When a Teensy update happens it is on a timeline of many things - typically driven by an Arduino Release getting it ready when users will start needing it - not a casual collection of stuff. And to be included it has to be from a current easy to include Pull Request that won't result in breakage or support issues. It is your code and if you can't see fit to make it right and easy - you can't expect someone else to drop something to fix it and test it to be right, no doubt HID is something Paul knows and works with - but unless he has the hardware and time to test it is unreasonable to expect him to do 'your' work as well as his own.

This is built into a monitor, right?

If I wanted to dig into this (and I do) and if I had time to really investigage (that part is questionable), exactly which product would I need to buy or borrow, so I could watch its USB traffic?

If you are referring to this post in the other thread - that I was referring to - Paul asks about hardware needed to repro and test - the word NOT is nowhere to be found. In fact quite the opposite - he wants to address it and wants to know what hardware he'd need to get on hand to do that.

I posted a quote from less than 2 weeks back - I scanned the thread and didn't see an update - your reply only reinforces my point - if you are evolving work and improvements on this software there is no way the OLD code for a PULL request would be considered as it has been improved and bug fixed or whatever. Not only may it not integrate with other changes - but it hasn't been tested and is months out of date.
 
Teensyduino's HID code is effectively in a feature freeze until at least the 2 new products in development now are fully released. I'll very likely also put some real work into the website (perhaps a few months or longer) before considering any HID changes. This is one of many HID features requested. In fact, there's at least a few different requests for touchscreen emulation, one with descriptors for multi-touch. I do save a list of links and notes, even if it takes me months or years go get back to them.

Even when I do work on HID again, which might be many months away, I must say the combative tone of these messages gives me pause. In the very early days of Teensy 1.0, a friend of friends in the local community wanted to collaborate on AVR USB development. He consistently used a similarly aggressive tone. While he did make some small contributions and offer some interesting ideas (at least early on), in the end his involvement turned out to be a huge time sink, as well as being terribly draining. It actually dragged on for a few years. While this situation might not be analogous, I'm really not eager to repeat and relearn that painful lesson.

The last thing I want to do is practice censorship or impose self-censorship. I really do appreciate blunt honesty, even when the truth is unpleasant. But when it comes to accepting contributions and collaboration, which necessarily brings a level of personal involvement, I do have some limits. The angry-sounding rhetoric in this thread is a disincentive. I've been burned before. If possible, please tone this down a bit? If not, if heated language is simply your style, maybe simply forking the code or contributing to a different project would be best for everyone?
 
Hello Paul

I am very suprised to read from you after 9 months.

If you have read my first posting above you will notice that there is absolutely nothing "combative" or "aggressive" in it.
You have never answered to it.

If you read my second posting above you will see that there is nothing aggressive in it.
You have never answered to it.


So if you say that you did not answer to this thread because of my "angry-sounding rhetoric", this is definitely NOT true.
In the beginning there were two very technical postings which you surely have read but you did not give any feedback.

Obviously it is frustrating when you do a contribution for another one's project and you get completely ignored.

You are the one who earns money with Teensy.
I invested my time to give a completely free solution for a problem in a project that is not mine.
And you did not show interest.
That is frustrating.

My code is a benefit for you because it makes your product more interesting and it solves a problem that many people encounter.
Relative mouse movement is usless for many use cases.

___________________________

OK.
Let's forget the past and try to start a new conversation without emotions.

I have implemented 3 very different touch screen descriptors of which the user can chose one by setting a compiler switch.

1.) A pointer descriptor that is a copy of real touch screen from ELO.
2.) A digitizer descriptor that uses a pen.
3.) A multi-touch descriptor that uses one finger.

All 3 descriptors comply the HID standard.

I tested them where I could:
They work all on Windows XP, 7, 8, 10.
They also work all on Ubuntu and Knoppix.

I cannot test them on Android but a Codeproject user wrote that the descriptors 2.) and 3.) work also on Android.
What is missing is the feedback from a OSX user.

My article on Codeproject has 11000 views and nearly 300 downloads but it is very difficult to get people to contribute.
They download the project and you hear nothing more from them.

But if there is a will to implement this into Teensy there will be surely no problem to find someone here in this forum who can test on a MAC.

___________________________

In my humble opinion the files usb_desc.h and usb_desc.c should be rewritten.
You have hardcoded the HID descriptors in compiler #define's.
This makes the HID stuff very inflexible.


The problem that arises is that you must chose in the Arduino compiler either


"Keyboard/mouse/joystick"
or
"Serial/Keyboard/mouse/joystick"


But there is no possibility to chose

"Keyboard/mouse/touchscreen"
or
"Serial/Keyboard/mouse/touchscreen"
or
"Serial/Keyboard/mouse/touchscreen/joystick"


And as there are 3 different touchscreens it would even have to be:

"Serial/Keyboard/mouse/touchscreen1/joystick"
"Serial/Keyboard/mouse/touchscreen2/joystick"
"Serial/Keyboard/mouse/touchscreen3/joystick"


As I did not want to rewrite everything I let the user chose by compiler switch what he wants.

I think that these options should be removed from the menu in the Arduino compiler and there should be one single file ONLY for configurations, like for example

settings.h

where the user can define what he wants to be included into the HID descriptor (and other settings more).

In this settings.h the user should select if he wants serial, keyboard, mouse, touchscreen, joystick or any combination of them.
This file should have very detailed comments that describe each of the compiler switches.

This should not be in the Ardiuno compiler's menu because it is too inflexible.
__________________________

So please tell me what is the future of this HID stuff.
Will you implement touchscreen support one day ?

I offer you my help if it is needed.
I'am programming in C / C++ since 30 years so you don't have to fear to get crappy code from me.

Elmü
 
Last edited:
I like the idea of being able to choose which HIDs you want but this issue is they need to be set as compiler flags as they get compiled before your program does.

Having a settings.h breaks away from the Arduino flow and make it's complicated for newbies who remain within the environment.

Perhaps if we could find a nice solution to do this from within the IDE we'd be onto a winner but I'm having trouble thinking of a way to do this
 
This is on my list of HID stuff to merge.

Please understand HID (and many other things) are a feature freeze at this time, and will remain so for many months.
 
Hello Xenoamor

> I like the idea of being able to choose which HIDs you want but this issue is they need to be set as compiler flags as they get compiled before your program does.

This is not correct.
I studied the Teensy code.
It would be possible to build the HID descriptor dynamically in C code.

In the file usb_dev.c you find a function
void usb_setup(void)

It would be possible to build the descriptor dynamically instead of using hardcoded descriptors.

For example the hardcoded switches NUM_ENDPOINTS and NUM_INTERFACE could be set by C code dynamically depending on the amount of devices to be added.
You define a struct and set the values according to devices to be included.

A new function could be written that adds only the HID devices that the user wants be included.
But to obtain this flexibility you would have to rewrite everything.

____________________________________

> Having a settings.h breaks away from the Arduino flow and make it's complicated for newbies who remain within the environment.

I agree with you.

An additional problem for newbies that must not be forgotten is this:

Windows stores the USB devices in the Registry.
When you change the composition of the HID devices in the descriptor (for example using touchscreen2 instead of touchscreen1) and reconnect the Teensy, Windows will not recognize the changes because they are stored only in the Registry when the device is connected for the first time.

You must uninstall the device before making changes.
Or you must use another ProductID for each configuration.

In my current code on Codeproject I use

0x048A for Keyboard/Mouse/Touchscreen1
0x048B for Serial/Keyboard/Mouse/Touchscreen1
0x048C for Keyboard/Mouse/Touchscreen2
0x048D for Serial/Keyboard/Mouse/Touchscreen2
0x048E for Keyboard/Mouse/Touchscreen3
0x048F for Serial/Keyboard/Mouse/Touchscreen3

This allows to change the composition of the HID descriptor without having to uninstall the driver.

The joystick is disabled, because I don't need it, but it can easily be enabled.


My proposal for the final solution is to use one product ID and put ALL possible devices into it.

For example:
0x0491: Serial + Keyboard + Mouse + Touchscreen1 + Joystick.
0x0492: Serial + Keyboard + Mouse + Touchscreen2 + Joystick.
0x0493: Serial + Keyboard + Mouse + Touchscreen3 + Joystick.

If the user does not need Serial or Joystick it will be present in the descriptor but he simply does not use it and it makes no harm.

The switching between the 3 touchscreens can then either be done in the menu of the Ardiono compiler or in a compiler switch.

_______________________________________


And there is another thing that must not be forgotten:

If a settings file is used, the Teensy installer must not overwrite the user's changes.
This can easily be done by calulating a CRC of the entire file and when it differs, a message box is shown:

"You have modified the file settings.h. Your old file has been stored as settings.bak. Please adjust the new file with your desired settings."

This is also not very user friendly, but the current solution (having a hardcoded menu in the Arduino compiler) is very inflexible.

_______________________________________

It seems the perfect solution does not exist.
At the end Paul has to decide how to make it user friendly and at the same time more flexible for future device additions.

_______________________________________

P.D.
Paul has implemented a workaround for Macintosh that adds a border of 7.5% around the usable area.

if (mac) {
// ugly workaround for Mac's HID coordinate scaling:
// http://lists.apple.com/archives/usb/2011/Jun/msg00032.html
usb_mouse_offset_x += 161061273ul;
usb_mouse_offset_y += 161061273ul;
usb_mouse_scale_x = (1825361101ul + (width >> 1)) / width;
usb_mouse_scale_y = (1825361101ul + (height >> 1)) / height;
}

I suppose that this workaround can now be removed with the new touchscreen descriptos.
Someone has to test that on a Macintosh.

_______________________________________

P.D.D:

Please note that I completely rewrote the Mouse class.

1.) The code is cleaned up (cryptic stuff like the above lines has been replaced with understandable code)
2.) The Mouse.moveTo() command allows to pass either absolute coordinates in percent values (0.00% ... 100.00%) or in screen coordinates.
3.) The Mouse.move() command allows relative coordinates of 16 bit. (movements of -127 ... +127 in the current mouse class is not enough for many cases)


Elmü
 
Last edited:
It would be possible to build the descriptor dynamically instead of using hardcoded descriptors.

Hi Elmue,
I'm struggling to imagine a method of doing this without storing all the descriptors in flash memory. Can you perhaps post a crude example? I really don't have much experience with this library stuff
 
Hello

Example: Instead of:

Code:
static uint8_t config_descriptor[CONFIG_DESC_SIZE] = 
{
        9,                       // bLength;
        2,                       // bDescriptorType;
        LSB(CONFIG_DESC_SIZE),   // wTotalLength
        MSB(CONFIG_DESC_SIZE),
        NUM_INTERFACE,           // bNumInterfaces
        1,                       // bConfigurationValue
        0,                       // iConfiguration
        0x80,                    // bmAttributes  (NOT self powered, no wake up)
        50,                      // bMaxPower

        .........

#ifdef MIDI_INTERFACE
        .........
#endif

#ifdef KEYBOARD_INTERFACE
        9,
        4,
        KEYBOARD_INTERFACE, 
        0, 
        .........
        KEYBOARD_SIZE, 0, 
        KEYBOARD_INTERVAL,
#endif


you could first fill structs like this with data:

Code:
struct kDescriptor
{
    byte   bLength;
    byte   bDescriptorType;
    byte   wTotalLengthLow;
    byte   wTotalLengthHigh;
    byte   bNumInterfaces;
    byte   bConfigurationValue;
    byte   iConfiguration;
    byte   bmAttributes;
    byte   bMaxPower;
};

And the code that fills in the descripor data would look like this:

Code:
void FillAllDescriptors(byte* pBuffer)
{
    .......

    #ifdef MIDI_INTERFACE

         kDescriptor* midi = (kDescriptor*)pBuffer;
         midi->bLength         = 9;
         midi->bDescriptorType = 4;
         ......

         pBuffer += sizeof(kDescriptor);

    #endif

    #ifdef KEYBOARD_INTERFACE

         kDescriptor* keybd = (kDescriptor*)pBuffer;
         keybd->bLength         = 9;
         keybd->bDescriptorType = 4;
         ......

         pBuffer += sizeof(kDescriptor);

    #endif

    etc...

}

As you see the code stays in #ifdef #endif blocks which means that if you turn off the Midi interface (by not defining MIDI_INTERFACE) the code that fills in the Midi data does not exist. So you would not consume memory for USB devices that you do not use.

But as I said before: To make the code more dynamic, you would have to rewrite everything.

Elmü
 
Last edited:
Status
Not open for further replies.
Back
Top