USB mouse/touchscreen with Teensy3.2 for Android

Status
Not open for further replies.

jz11

Member
Hello,
here is a brief description of the setup:
7" capacitive touch screen with a FT5316 controller talking to Teensy via i2c, the function returns coordinates from the FT5316 as x=0~800, y=0~480, so far I'm not trying multitouch, but the controller is capable of it, just single touch.

I use arduino ide 1.8.5 with teensyduino 1.40
I've tried all possible combinations of USB types selected in the IDE, and I've tried the absolute mouse approach and the touchscreen class, couldn't get the touchscreen class to work with Android (tried 6.0 and 7.0) at all, it sort of did work with Windows7, but it was registering single clicks even though I was holding my finger down, and generally not behaving like I expected it to at all, setting it up as a mouse has gotten me the furthest so far.

this code lets me draw stuff in Windows paint like I expect it to, but once plugged in into the Android device - nothing happens, at best the the cursor pops up in the middle of the screen, but clicks are not registered, nor there is any "movement", meaning - clicking on different parts of the touch screen gives the same result - nothing happens. The method of calculating XX/YY values to be used for Mouse.moveTo is crude, but this was just for general testing, I didn't try to refine it since it doesn't do what I need to on the Android device.
Code:
#include <SPI.h>
#include "Wire.h"
#include "FT5xx6.h"

#define CTP_INT           2    // touch data ready for read from touch panel
FT5xx6 cmt = FT5xx6(CTP_INT);

word prev_coordinates[10]; // 5 pairs of x and y
byte nr_of_touches = 0;

int isLeftPressed = 0;
int didMouseMove = 0;


void setup() {
  cmt.init(true);
  Mouse.begin();
}

void loop() {
    byte registers[FT5xx6_NUMBER_OF_REGISTERS];
    word coordinates[10];
        
      if (cmt.touched()){
        cmt.getRegisterInfo(registers);
        nr_of_touches = cmt.getTouchPositions(coordinates, registers);
        //singleTouchOutput(coordinates);
        int doImoveMouse = 1;
        word x = coordinates[0];
        word y = coordinates[1];
        word XX = 12.5*x ;
        word YY = 20.8*y ;
            
        if (y > 2000 || y <= 0 || x > 2000 || x <= 0 ){
          doImoveMouse = 0;
        }
          
        if (doImoveMouse == 1){
          Mouse.moveTo(XX, YY);
        } 
            
        if (registers[2] == 1 && isLeftPressed == 0){
          Mouse.press(MOUSE_LEFT);
          isLeftPressed = 1;
        }
        
      } 

      if (registers[2] == 0 && isLeftPressed == 1){
        Mouse.release(MOUSE_LEFT);
        isLeftPressed = 0;
      }
      delay(25);
}

So I was googling around for solution, and red these topics and tried everything that was in there:
https://forum.pjrc.com/threads/32331-USB-HID-Touchscreen-support-needed
https://forum.pjrc.com/threads/2894...ice-to-Teensyduino-1-23-(works-also-on-Linux)
I also tried the older IDE with the older teensyduino and edited usb descriptor files by mr.Elmue, but couldn't get it to work like it should with the android device. https://www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with

Then I tried to figure out the usb descriptors, but the official text is basically unreadable for an amateur like me, but I opened up the usb_desc.c from teensy/avr/cores/teensy3/ and had a look around, and as soon as I recompiled my sketch with these changes, the thing started exhibiting some life when plugged into the Android device:
Code:
#ifdef MOUSE_INTERFACE
// Mouse Protocol 1, HID 1.11 spec, Appendix B, page 59-60, with wheel extension
static uint8_t mouse_report_desc[] = {
       // 0x05, 0x01,                     // Usage Page (Generic Desktop)  <---- I commented out these 2 lines
       // 0x09, 0x02,                     // Usage (Mouse)                         <---- and added 2 first lines from the touch multitouch descriptor
        0x05, 0x0D,                     // Usage Page (Digitizer)
        0x09, 0x04,                     // Usage (Touch Screen)

Android device started to actually react to me touching the touchscreen, it is registering a click and a movement, but the movement is relative, and I tried editing few other lines in that same mouse descriptor section that had some meaningful comment, but nothing seemed to result is anything resembling an absolute coordinate system, at best nothing happened, or the android just wouldn't respond to recompiled sketch with additional modifications to descriptor file at all.

Am I doing something really wrong or there are some known problems that I wasn't able to google solutions for?
 
Last edited:
So I fiddled around a bit more with Mr.Elmues work, and his touch descriptors #1 and #3 didn't work, #1 is detected by Android as mouse, some events can be observed via shell>getevent, but movement doesn't work, #3 is not detected at all, no events.
But by modifying the #2 I got so far as to move to absolute coordinates, mouse.press(mouse_left) works, but press-hold-drag doesn't work, and mouse cursor is being showed.
Under windows though, absolute movement doesn't work, but press/release still works.
Hopefully someone more knowledgeable can have a look at this and at least hint to potential fixes.
Code:
    #elif TOUCH_DEVICE == 1 // single touch device (product ID: 0x048C + 0x048D)
	// ================================================================================

        0x05, 0x0d,                    // USAGE_PAGE (Digitizer)
        //0x09, 0x02,                    // USAGE (Pen)
	0x09, 0x04,                    // USAGE (Touch Screen)
        0xa1, 0x01,                    // COLLECTION (Application)
		
		// declare a finger collection
        0x09, 0x20,                    //   Usage (Stylus)
        0xA1, 0x00,                    //   Collection (Physical)

        // Declare a finger touch (finger up/down)
        0x09, 0x42,                    //     Usage (Tip Switch)
	0x09, 0x32,                    //     USAGE (In Range)
        0x15, 0x00,                    //     LOGICAL_MINIMUM (0)
        0x25, 0x01,                    //     LOGICAL_MAXIMUM (1)
        0x75, 0x01,                    //     REPORT_SIZE (1)
        0x95, 0x02,                    //     REPORT_COUNT (2)
        0x81, 0x02,                    //     INPUT (Data,Var,Abs)

        // Declare the remaining 6 bits of the first data byte as constant -> the driver will ignore them
        0x75, 0x01,                    //     REPORT_SIZE (1)
        0x95, 0x06,                    //     REPORT_COUNT (6)
        0x81, 0x01,                    //     INPUT (Cnst,Ary,Abs)

        // Define absolute X and Y coordinates of 16 bit each (percent values multiplied with 100)
        // http://www.usb.org/developers/hidpage/Hut1_12v2.pdf
        // Chapter 16.2 says: "In the Stylus collection a Pointer physical collection will contain the axes reported by the stylus."
        0x05, 0x01,                    //     Usage Page (Generic Desktop)
        0x09, 0x01,                    //     Usage (Pointer)
        0xA1, 0x00,                    //     Collection (Physical)
        0x09, 0x30,                    //        Usage (X)
        0x09, 0x31,                    //        Usage (Y)
        0x16, 0x00, 0x00,              //        Logical Minimum (0)
        0x26, 0x10, 0x27,              //        Logical Maximum (10000)
        0x36, 0x00, 0x00,              //        Physical Minimum (0)
        0x46, 0x10, 0x27,              //        Physical Maximum (10000)
        0x66, 0x00, 0x00,              //        UNIT (None)
        0x75, 0x10,                    //        Report Size (16),
        0x95, 0x02,                    //        Report Count (2),
        0x81, 0x02,                    //        Input (Data,Var,Abs)
        0xc0,                          //     END_COLLECTION

        0xc0,                          //   END_COLLECTION
        0xc0                           // END_COLLECTION

        // With this declaration a data packet must be sent as:
        // byte 1   -> "touch" state          (bit 0 = pen up/down, bit 1 = In Range)
        // byte 2,3 -> absolute X coordinate  (0...10000)
        // byte 4,5 -> absolute Y coordinate  (0...10000)
 
> I also tried the older IDE with the older teensyduino and edited usb descriptor files by mr.Elmue, but couldn't get it to work like it should with the android

I never tried Android but people reported on Codeproject that it works on Android.

A user wrote:
> I applied these examples in Android and I found Android supports multi-touch only.

It seems you did not read the comments section.

__________________________

> but press-hold-drag doesn't work,

This may be because your code is wrong.
If move works alone and press works alone then drag must also work!
Drag is only a combination of press, move and release.

___________________________

> Under windows though, absolute movement doesn't work,

Wrong.
Under Windows ALL my descriptors work perfectly.

My article says:
> I tested my project on Windows XP, 7, 8, 10 and on Linux (Suse, Ubuntu, Knoppix) where it works seamlessly.

If it does not work you, your controlling code is wrong.

_________________________

The use of the variable doImoveMouse does not make any sense.
You set this variable to 1 even if you do NOT move the finger.
You don't even check if the coordinates have changed since the last call to loop().
And the variable didMouseMove is never used.
This looks to me like a quite sloppy code.

Every 25 ms you send data to the poor Android device that is flooded with always the same coordinates.

To see what your code is exactly doing I recommend strongly to print debug output with Serial.println() and check in the Serial monitor the data that you are exactly sending.
 
Hi Elmue, don't think of my posts like a criticism of your work, the Android device I tried your exact code on resulted on with descriptors #1 and #2 being detected as a mouse, I see some activity with getevent command in shell, but it doesn't register absolute movement or clicks not being processed, something like that, #3 is being detected as touchscreen and there is absolutely nothing in getevent when I touch the sensor panel. #2 was the closest (detected as mouse) to what I thought of being correct, and with some tweaks in the modified #2 descriptor I posted above, it started working, sort of.
The drag-and-drop is odd, when I longpress on an icon on the desktop, and drag my finger on the sensor to where the delete/uninstall icons are, I can delete/uninstall that app, but the animation of the icon being dragged doesn't show, also in navigation software, when I use real usb mouse, click and hold somewhere on the map, then move the mouse, I can move the map around, but when I do the same thing with the code above and touchscreen, then all I get is cursor being moved on the map, but the map is not "picked up" and moved.
This leads me to think that there are peculiarities regarding how the data sent is being interpreted by the Android device driver, that is compiled with the kernel there, and probably the people from codeproject comments reporting your descriptors working with their android had different mouse driver, or maybe some additional touch screen driver that I simply don't have in the build that is in my device.
And the modified descriptor #2 that worked on my Android doesn't work on windows indeed, well, it being detected, and clicks are being registered, but not the absolute movement events, so in paint I can touch the screen, drag the finger, and nothing happens, but when I touch the screen and move my regular mouse with the other hand, then then I can see that the left-press was indeed registered.

doImoveMouse was there to catch a bug with the ft5xx6 driver, where sometimes on the touch or release the coordinates are being interpreted wrong and I get some really large numbers, so I simply wanted to eliminate those events, it is all it is there for.

Also, when I do the getevent on the Android, and touch and hold the screen, the androids getevent doesn't show repeated flooding of the same movement coordinates, the event log stays still till I actually move my finger or release it, I'm not arguing that the code above is not flooding the android with repeated data, so I didn't really think of that as a problem before, but now that you mentioned it, I'll implement a filter to not send anything over the usb, if the state on the touch screen hasn't changed. My sketch wasn't perfect, it was only the start to get things happen somewhat, and now that they do, I can start polishing it, that was my line of thought.

thanks a lot for the previous work you've done, it has helped a lot to get some understanding of how the usb descriptors work.
There was another good site I found, with a link to a descriptor build and validation tool - http://eleccelerator.com/tutorial-about-usb-hid-report-descriptors/ - this is what helped me to get some understanding of how to actually read them, because the info on the usb.org is like chinese to me...
 
Last edited:
Hello jz11

In your last answer you explain the problem much better.
Intead of just saying "Drag does not work" you explain the problem more detailed.
In this case I also suppose that the driver is not very well programmed and does not work with that specific descriptor.
Very weird.

I suppose the reason is simply that the descriptor is a pen.
And with a pen you draw lines but with a pen you never do drag operations.
Therfore you must use the USAGE(Finger) instead of USAGE(Pen).
But it will not be enough to simply replace Pen with Finger.
Sadly USB is much more complex, documentation and examples are sparse and drivers have lots of bugs.


You are only talking about descriptors #1, #2, #3.
Did you ever try descriptor #4 which has been reported to work an Android?


If I understand you correctly it still does not work 100% yet?


In this case the fastest way to get this to work would be if you could buy any hardware USB touchscreen that works on Android correctly and just copy it's descriptor. If you get it working 100% finally you could send me the descriptor for Android and I publish it on Codeproject because a lot of people are asking for that.


But it would teoretically also be possible that the touchscreen support in Android is simply buggy and you cannot do anything to get it work.
(The built-in touchscreen is surely not USB)
You can try to file a bugreport posting your descriptor and explaining the problem. Maybe any Android developer answers you.


> and probably the people from codeproject comments reporting your descriptors working with their android had different mouse driver.

You could try to get an answer from them.
 
I think I'm starting to understand where the problem is, I took another look at the Android device and the getevent output
here is a link describing it - http://newandroidbook.com/Book/Input.html?r

If I read this correctly, then when I plug the programmed teensy into the android device, it is being detected as 3 devices:
event5, event6 and event7

and when I touch the screen, the commands from the teensy don't seem to originate from one device, but rather 2, which is confusing the android os into thinking that clicks/holds etc come from one device, and absolute movement coordinates come from a different device - hence the presses and releases sort of work, the movement works, but android doesn't think that the individual press and consequent movement originated at one device, so there is some confusion regarding the drag-and-drop operation.
Code:
add device 6: /dev/input/event5
  name:     "Teensyduino Keyboard/Mouse/Touchscreen"
could not get driver version for /dev/input/mouse2, Not a typewriter
add device 7: /dev/input/event6
  name:     "Teensyduino Keyboard/Mouse/Touchscreen"
could not get driver version for /dev/input/mouse3, Not a typewriter
add device 8: /dev/input/event7
  name:     "Teensyduino Keyboard/Mouse/Touchscreen"
/dev/input/event6: EV_MSC       MSC_SCAN             00090001
/dev/input/event7: EV_KEY       BTN_DIGI               DOWN
/dev/input/event6: EV_KEY       BTN_MOUSE            DOWN
/dev/input/event7: EV_ABS       ABS_X             00000ad7
/dev/input/event6: EV_SYN       SYN_REPORT           00000000
/dev/input/event7: EV_ABS       ABS_Y             000018f1
/dev/input/event7: EV_SYN       SYN_REPORT           00000000
/dev/input/event7: EV_ABS       ABS_X             00000af0
/dev/input/event7: EV_ABS       ABS_Y             00001906
/dev/input/event7: EV_SYN       SYN_REPORT           00000000
/dev/input/event7: EV_ABS       ABS_X             00000ad7
/dev/input/event7: EV_ABS       ABS_Y             0000191b
/dev/input/event7: EV_SYN       SYN_REPORT           00000000
/dev/input/event7: EV_ABS       ABS_X             00000ae3
/dev/input/event7: EV_ABS       ABS_Y             00001906
/dev/input/event7: EV_SYN       SYN_REPORT           00000000
/dev/input/event6: EV_MSC       MSC_SCAN             00090001
/dev/input/event6: EV_KEY       BTN_MOUSE            UP
/dev/input/event6: EV_SYN       SYN_REPORT           00000000

when I plug in just a regular usb mouse and do the same thing, then all the events originate from the same device, and the drag-and-drop works like it should
here is the getevent report:
Code:
/dev/input/event4: EV_MSC       MSC_SCAN             00090001
/dev/input/event4: EV_KEY       BTN_MOUSE            DOWN
/dev/input/event4: EV_SYN       SYN_REPORT           00000000
/dev/input/event4: EV_REL       REL_Y               00000001
/dev/input/event4: EV_SYN       SYN_REPORT           00000000
/dev/input/event4: EV_REL       REL_Y               00000001
/dev/input/event4: EV_SYN       SYN_REPORT           00000000
/dev/input/event4: EV_REL       REL_Y               00000001
/dev/input/event4: EV_SYN       SYN_REPORT           00000000
/dev/input/event4: EV_REL       REL_X               00000001
/dev/input/event4: EV_SYN       SYN_REPORT           00000000
/dev/input/event4: EV_REL       REL_Y               00000001
/dev/input/event4: EV_SYN       SYN_REPORT           00000000
/dev/input/event4: EV_MSC       MSC_SCAN             00090001
/dev/input/event4: EV_KEY       BTN_MOUSE            UP
/dev/input/event4: EV_SYN       SYN_REPORT           00000000

this leads me to think that something is wrong with the usb code, it is not just a descriptor problem, the thing is that since I started learning writing code from arduino ide, which is sort of higher level code, it is quite difficult for me to understand the lower level usb related code, but now that the issue is sort of narrowed down somewhat, I'll have a longer look at it and try to find out why some Mouse. commands go to one device, and some to another...
 
And a thought regarding the USB Type:
it might be not a bad thing to be able to select from the base code which USB option to include in the device, for instance, this project of mine will just a touch screen, but the drop down menu in the Arduino IDE for Teensy doesn't allow to select just one single touch screen, keyboard, mouse etc is together with it, and might compound the issue...
 
You are surprised that there are 3 USB devices?
The title of my article says:

A USB HID Keyboard, Mouse, Touchscreen emulator with Teensy

Obvioulsy these are 3 USB devices.

_______________________

> and when I touch the screen, the commands from the teensy don't seem to originate from one device,

Then there is something wrong in your code.
This is surely the reason that drag does not work.

After looking your first post I see now that you use:
Mouse.press(MOUSE_LEFT);
and
Mouse.moveTo()

This is definitely wrong.
As the name already says: MOUSE_LEFT is a mouse button.
You send the press event to the mouse and the move event to the touch screen.


My article explaines this even with examples:

The command Mouse.move() simulates RELATIVE movements with the mouse device.
The command Mouse.moveTo() simulates ABSOLUTE movements with the touchscreen device.

If you mix the devices a drag operation will not work.

To do a drag/drop operation use:

moveTo() to define the absolute coordinate (drag = start position) where the finger goes down
and then
set_buttons(0, 0, 0, 1) to define the finger of the TOUCHSCREEN is down now
and then
moveTo() to drag the finger to the specified absolute coordinate (drop = end position)
and then
set_buttons(0, 0, 0, 0) to release the finger.

Have a look into the sourcecode (usb_mouse.h and usb_mouse.c) to understand this.

I don't know how Android behaves but it may be necessary to do the drag movement in multiple steps.
For example:

moveTo(10,10);
delay(5);
moveTo(20,20);
delay(5);
moveTo(30,30);
delay(5);

etc..

_______________________

> it might be not a bad thing to be able to select from the base code which USB option to include in the device, for instance, this project of mine will just a touch screen,

This is completely correct.
I also complained to Paul that this is not very flexible.
I tried to change that but it would be so EXTREMELY complicated that it would mean to rewrite everything and I gave up.

If there is a keyboard device that you don't use it will make no harm.
So this is not really a problem.

But to change that I would have to change several header files from Paul's Teensyduino library
This would be a huge work and as long as Paul does not integrate these changes into his Teensydiuno library it would be a work in vain.


The reason why Paul's library defines the USB devices in form of C compiler switches is that this is the only way to store the USB descriptors precompiled in the EEPROM of the Teensy. But it would be possible to design a more flexible code that builds the USB descriptors on the fly when the host computer asks for them. It would be possible to define in the sketch which devices to expose to the computer. But this would mean to rewrite everything and this work must be done by Paul. The drawback would be that the code becomes longer and more EEPROM is used for code.
 
Last edited:
Well. You don't reply anmore.
This shows several things:

SUMMARY:
- Your problem had nothing to with Android.
- You made a severe programming error in your sketch.
- The reason for this error was that you did not read my article thoroughly.
 
Last edited:
I didn't have free time to develop this further yet, but I'm going to get this working eventually and post the end result

couple things I did check, mouse.set_buttons didn't fix the issue of click events and movement events origin, I was going back and forth testing different versions of android ide and teensyduino, and found that (afair) the latest one had some really weird behavior, where mouse.press(left_button) resulted as 'up' or 'lift' event, and mouse.release as 'down' - basically the opposite of what it should be, the very same sketch with mouse.set_button(1,0,0,0) did work as expected and result in the 'down' on the android device, and 0,0,0,0 was a 'up'.

I did dig up an old project, where I had atmega32u4 as a "mouse" with resistive touch screen attached to it, the code originated from a Russian carpc forum site, but the usb portion seems to be some version of teensy usb code that has been modified, I'll attach the source code so anyone interested can have a look - when this hex is written to the 32u4 it is acting like a usb mouse, drag and drop works like it should, events originate from the same device etc, the only "problem" is that the android is showing the mouse cursor wherever you touch, which is a bit ugly.

here is the latest sketch I was using with the teensy, one note regarding the code, it is silent (no messages on the usb) if I don't touch the screen, when I do, the interrupt is being activated by the touchscreen controller and teensy begins to read registers, as long as I touch it and don't move my finger, the usb event log stays still, so there is no flooding going on as far as I can tell, I put that one safeguard there just in case, but I think it might be turned later into a filter, where small movements would be filtered out, to reduce the traffic on the usb.

Code:
/***************************************************************************************
 Connections:

 ER-TPC070-6     teensy3.2

 1                           GND
 2               18          SDA
 3               19          SCL
 4                           RESET
 5               2           INT
 6                           NC
 7               3.3V        VDD
 8                           VDD
 9                           GND
 10              GND         GND

****************************************************************************************/
#include <SPI.h>
#include "Wire.h"
#include "FT5xx6.h"

#define CTP_INT           2    // touch data ready for read from touch panel
FT5xx6 cmt = FT5xx6(CTP_INT);

byte nr_of_touches = 0;
int isLeftPressed = 0;
word  previous_x = 0,
      previous_y = 0;
      
void setup() {
  cmt.init(true);
  Mouse.begin();
}

void loop() {
    byte registers[FT5xx6_NUMBER_OF_REGISTERS];
    word coordinates[10];
        
      if (cmt.touched()){
        cmt.getRegisterInfo(registers);
        nr_of_touches = cmt.getTouchPositions(coordinates, registers);
        
        int doImoveMouse = 1;
        
        word x = coordinates[0];
        word y = coordinates[1];

        if (x == previous_x && y == previous_y){ // prevent repeated coordinates being sent over the usb
          doImoveMouse = 0;
        }
            
        if (y > 2000 || y <= 0 || x > 2000 || x <= 0 ){
          doImoveMouse = 0;
        }
          
        if (doImoveMouse == 1){
          word XX = 12.5*x ;
          word YY = 20.8*y ;
          Mouse.moveTo(XX, YY);
        } 
            
        if (registers[2] == 1 && isLeftPressed == 0){  // report the 'down' only when the touch screen controller has successfully registered a touch event, and do that only once
          Mouse.press(MOUSE_LEFT); // tested with Mouse.set_buttons(1,0,0,0) as well, result was the same, 'click' event reported from a different usb device, not the same as Mouse.moveTo messages come from
          isLeftPressed = 1;
        }

        
        
      } 

      if (registers[2] == 0 && isLeftPressed == 1){
        Mouse.release(MOUSE_LEFT);
        isLeftPressed = 0;
      }
      
      delay(25);
}
 

Attachments

  • FT5xx6.cpp
    3.7 KB · Views: 183
  • FT5xx6.h
    2.1 KB · Views: 136
  • resistive_32u4_TouchController.zip
    218 KB · Views: 163
Russian project ???
You are searching at the completely wrong place.

You are incredible!
It becomes obvious that you did neiter read my article nor did you read my answer from second january here.

YOUR CODE IS STILL WRONG !

> didn't fix the issue of click events and movement events origin

OBVIOUSLY !
Because your code has STILL the same bug.


> Mouse.press(MOUSE_LEFT);
> // tested with Mouse.set_buttons(1,0,0,0)

BOTH are wrong !

Is it really so difficult to read THOROUGHLY what I wrote ????

Just fix your bug and it will work.


P.D.
A programmer who is not able to read a documentation is not a programmer.
 
Elmue, I made a mistake when I wrote in my post mouse.set_button(1,0,0,0), I meant the 0,0,0,1 , it still didn't work, the DOWN/UP event was registered as a different event by the android and the drag and drop didn't function.

Yesterday evening I got the same sketch (that you said is the cause of the problem) to work with 32u4 by modifying 1.6.7 arduino ide mouse.cpp and mouse.h files (these were a bit more easier for me to understand than the uncommented teensy usb related stuff), I made a custom descriptor based on the default mouse descriptor, but changed the button down-up portion to work with absolute coordinates, interesting thing is that the android driver is working properly with this descriptor only if there are BOTH "tip switch" and "in range" bits sent along the ABS coordinates, if the descriptor is made with only "tip switch", then the driver doesn't register the coordinates as absolute, only relative, there were other interesting effects as well, I could make it work like a classic touch pad on a laptop keyboard, move the cursor around relative to its current position by dragging finger over the touch screen, double clicks worked to activate icons, but drag and drop requires a bit more work, I'm guessing that there needs to be smarter way of handling "in range" and "tip switch" bits, right now I'm just sending 11 for both of them whenever a touch even has occurred, and it seems to work fine.

Anyway, I attached the working arduino leonardo (32u4) sketch and the rest of the stuff needed to compile it with 1.6.7 version of the IDE (I didn't try with the newer one yet, may update the post when/if I do).

Note that is was made to work ONLY for Android (tested on couple different 6.0.1 images I had for Khadas VIM), under Windows it does some weird stuff which I couldn't care less about. I also tested it on a virtual machine running Ubuntu 12, it does seem to work, absolute coordinates work, clicks work, drag and drop as well, but the 127x127 resolution is probably too little for larger screens.
 

Attachments

  • FT5xx6.h
    2.1 KB · Views: 185
  • Mouse.h
    1.9 KB · Views: 154
  • Mouse.cpp
    4 KB · Views: 160
  • FT5xx6.cpp
    3.7 KB · Views: 139
  • _32u4_FT5316_capacitive_touch.ino
    2.9 KB · Views: 127
so I did some more investigating with a fresh installation of latest Arduino IDE (1.8.5) followed by latest teensyduino right, then made this simple sketch:
Code:
void setup() {
TouchscreenUSB.begin();
delay(5000);
}

void loop() {
TouchscreenUSB.press(0, 200, 200, 64);
delay(100);
TouchscreenUSB.release(0);
while(1);
}

And the results from "getevent -i" command output are in the attached txt file, and it seems there is some sort of bug there with how the descriptor is being sent, because I'm pretty sure that is not what it is supposed to look like (and obviously it doesn't function, no events are being registered at all if I touch the screen), I also attached a picture showing how the descriptor from my previous post looks in android/linux just for a comparison. Note I did not edit any of the library files, this is compiled using all the default stuff.

working_usb_touch_descriptor.PNG
View attachment output.txt
 
@jz11:

> and it seems there is some sort of bug there with how the descriptor is being sent,

The descriptor in your screenshot is not the one from Teensyduino.
I have no idea where it comes from.

But how do you think to install Teensyduino and run the code on a Leonardo ???
Teensyduino is for Teensy boards ONLY.
 
I don't plan to run Teensyduino code on Leonardo or vice versa, that screenshot was just an example of what a log file output looks like on a working device (I made that descriptor up myself, loosely based on the default mouse descriptor), and the txt file contained the Android output from Teensy 3.2 programmed with latest Teensyduino set up as Keyboard+Touchscreen (which didn't work on Android), just for comparison.

Thanks Paul for the heads up on the updated files, I did some testing with them, and they work on Windows and Linux (Ubuntu in my case), but I ran into the same issue with Android again (not a problem any more, since I solved it with 32u4 and Arduino modified usb libraries I posted previously), so right now while my memory is still fresh from debugging the Leonardo code, I'm trying to understand what really is not working as it should with the teensyduino code and Android.

Please be aware that I'm a total novice in this sort of debugging, I'm no programmer etc, I'm just trying to figure things out, so if I'm doing something really wrong here, tell me, and maybe point in somewhat more correct direction. I don't expect anyone to educate me here or fix things for me, I'm just trying to push myself into learning new things.

As far as I can tell, Android is expecting something to be sent when the MULTITOUCH_INTERFACE is defined, and it doesn't receive it, and as a result it doesn't load the descriptor, but that seems to be Android issue only, since the same sketch works fine with Windows and Ubuntu.
Right now I'm trying to make sense of the usb_desc.h, and am trying to follow instructions in the comment at the begging on how to add a new item under the IDE>Tools>USB Type, plan was to make a single MULTITOUCH_INTERFACE device, so I can directly compare the android log file output with the Leonardo sketch that I got to work, to see where it hangs up. And I got so far as Android detecting the composite device, listing the HID device in the logs just like with the Leonardo, but that is it, then it stops (not hangs up, just no usb related activity). When compared to the Leonardo, the log file lists that it loads the descriptor and the thing becomes alive and the touchscreen begins to function.

So I dug into other teensyduino usb related files, enabled hardware serial in the sketch, hooked up ttl>serial converter on D0/D1 to see what is going on with the usb setup code in the usb_dev.c, and as far I see that there might be something missing in the usb_dev.c around line 585, that seems to deal with the setup of the USB device, when it is plugged in, there is a case 0x02:, which seems to load some addition stuff (which I can't figure out yet) when some interfaces are defined, and when I added this to it, I saw my debug messages pop up on the hardware serial, when I plug the thing into the Android device:
Code:
#ifdef MULTITOUCH_INTERFACE
		serial_print("does this do anything when I plug the thing in?");//
		serial_print("\n");//
#endif
 
after throwing a lot of spaghetti at the wall I narrowed down the problem to the descriptor (the latest updated usb_desc.c from the link posted by Paul), it fails to load only on Android 7 (Nougat), the older 6 (Marshmallow) Android seems to load the descriptor just fine (I didn't test the actual functionality yet, I was only exploring why the 7th one wouldn't detect the touchscreen device).
The actual problem with the descriptor is in the Usage (Contact Identifier) line, if that Usage is changed to something else, like Touch (0x33), the descriptor will actually load (and again I didn't test the functionality).

https://source.android.com/devices/input/touch-devices this has some information on how Android is classifying touch devices, so perhaps for the Contact Identifier to be parsed correctly, the X and Y have to be defined in some other way, basically - android may discard this descriptor because it doesn't fit the multitouch requirements, and doesn't fit single touch requirements as well, just a theory
 
You cannot change a Usage(Contact Identifier) into Usage(Touch)
This results in a corrupt descriptor that will NEVER work.
You did not understand at all what the descriptor is defining.
At the end it is irrelevant if Android loads your corrput descriptor or not. It will not work.


> https://source.android.com/devices/input/touch-devices this has some information on how Android is classifying touch devices

This document is talking in general.
It does not talk about descriptor details as contact identifiers.


> so perhaps for the Contact Identifier to be parsed correctly, the X and Y have to be defined in some other way,

You are surely looking in the wrong place.


Before making more nonsense changes in the descriptors you should read a very good manual about USB:
http://www.usbmadesimple.co.uk

Read this manual entirely until you understood how USB works otherwise you will only waste your time.

You cannot make changes in a descriptor that you don't understand.
 
Last edited:
Elmue, it might be a cultural issue or something, but your way of communicating with people (not just me) is really annoying, you presume to be an expert on this subject, when you clearly are not, if you tried to compile and run this with Android you would see that there clearly is a problem. I did not say changing the 0x51 to 0x33 will fix the issue, I explained the exact cause why this particular descriptor hangs up when being sent to Android Nougat. That is all, I'm not fixing EVERYTHING, I'm investigating why it doesn't work.

Also I went back to your solution, and the multitouch version (#2) of your touchscreen descriptor also contains the same Usage (Contact Identifier), and guess what - Android Nougat again doesn't load that particular descriptor. So any feedback you may have received on successful use of your code with Android devices must have been with some older version of it. So please add to your post that people may expect problems when it is used with Android 6 and Android 7 OS. I'm not even going into the issue that Android is parsing your touch descriptor and subsequent click messages as coming from 2 different devices - AGAIN - your code problem, not a sketch problem, because I posted above a solution with other descriptor and the same sketch works just fine!

Single touch descriptors are being loaded just fine on both versions of Android, so there must be some issue in the way the multitouch descriptor you used and Paul used is being written (or is constructed) or being transmitted to the Android device, because it is very clear now that the issue is related to how the Android is parsing the descriptor. The hardware serial debug I enabled shows that the whole descriptor is being sent to the Android device, but the v7 OS simply does nothing with it, I couldn't find a way to get any meaningful error message relating to this on the Android OS side ("logcat -v threadtime" doesn't say if there are any errors), and google searching has so far led to nothing.

Again - I'm not talking about a fix to something by getting rid of Contact Identifier, I understand what it is there for, I'm merely point out that the way it is implemented in current teensyduino libraries is not working with newer Android OS versions, and the attempt to pass a multipoint touch descriptor to the Android seems to be the cause. Again - talking only about use with Android devices, I already said earlier that it does load in Windows and Linux and seems to work.
 
For experimenting with the descriptor, please try just deleting the usage info and change the input from variable to constant.

For example, these are the 5 lines for the 7 bits of the contact identifier.

Code:
        0x09, 0x51,                     //     Usage (Contact Identifier)
        0x25, 0x7F,                     //     Logical Maximum (127)
        0x75, 0x07,                     //     Report Size (7)
        0x95, 0x01,                     //     Report Count (1)
        0x81, 0x02,                     //     Input (variable,absolute)

Just delete the first 2 lines, then change 0x02 to 0x01 on the last line, like this:

Code:
        0x75, 0x07,                     //     Report Size (7)
        0x95, 0x01,                     //     Report Count (1)
        0x81, 0x01,                     //     Input (constant)

Now technically speaking, this is telling the host we're going to send 7 bits that are constant and don't have any meaning. That's sort-of a lie, since the code will still put the 7 bits for contact identifier into the USB packet. But it's a pretty sure bet the host will just ignore these 7 bits. If you're *really* concerned, you could also edit the code to zero out all 7 of them, but I highly doubt that would many any difference.

The "Contact Identifier" appears to be a usage Microsoft made up. It's not listed in the HID usage tables published at usb.org. But then it could be pending for the next version? Who knows. Solid documentation on this stuff is hard to find. Microsoft documents it as this:

Specifies the identifier of the current contact. An identifier must remain constant while the contact is detected by the device. Each separate concurrent contact must have a unique identifier. Identifiers can be reused if a contact is no longer detected. If the device supports "in-air" packets (the contact is hovering above the surface), the identifier must persist from the time that the contact is detected until the time that it goes out of range. In the report descriptor in the EloMT sample, the comment for this usage is "Temp Identifier."

As you can see in usb_touch.c, I put in code to keep an 8 bit count of the previous contact ID. Each time a new touch is started, the number is incremented and stored in an array, so every report for that finger will keep the same contact identifier number until the last one is sent for release.

Why Android would not like this is a mystery....
 
Paul, that is what I did yesterday, I was going step by step through the descriptor to see which part causes it not to load, and as soon as I replace the 0x51 portion with a padding like you suggested, or keep the section and change the 0x51 to 0x33 for instance, then it is being loaded on the Nougat, I went back and fourth couple times changing it back to original and then replacing it to be 100% sure it is causing the issue.

Regarding the descriptor, I found a quite "fresh" one from 2011 here https://github.com/freebsd/freebsd/blob/master/share/misc/usb_hid_usages , it is from BSD though, not Android, but it does list Contact Identifier under the Digitizer page, just like it does in your descriptor (and Microsoft documents I've seen), so maybe the Nougat I have here has some sort of bug and cannot process that entry... I'm downloading the source code for it right now and will try to dig that list up in it, to make sure it understands that 0x51 usage at all. I'll come back and post my findings if I do.
 
New twist in the story - as we all know Androids may come in MANY different flavors, and some of them may have had their kernels compiled with single touch hid drivers only... turns out including multi-touch HID driver is an option, and I just have happened to luck into an image release by Khadas developers, that had the single touch version of kernel in their Nougat image, which led to this whole debacle (and the Marshmallow image from the same Khadas had that multi-touch enabled kernel, and that is why it had no problem parsing that descriptor). I also confirmed that by downloading 6.0 R3 and 7.1RC1 x86 Android images from https://www.osboxes.org/android-x86/ and running them on Oracles Virtualbox machines, and it works.

There is a way to load additional modules to a kernel for instance in case of that single-touch image I had, but from the instructions I've red it seems a bit messy, since that driver must be compatible with the kernel version used in that image, so the solution is not universal, if I'll find a way to do that easily enough, I will come back and add the info to this thread.

edit: as far as I've found out, there is no way of checking if the kernel in question is a single or multi-touch HID, /sys/modules may contain hid_multitouch file in it, or it can be a loadable module, and loadable modules may be in different places depending on the build configuration, and I couldn't find any universal command that would output exact list of loaded or loadable modules, so the test would be basically to plug the multitouch usb device in and see if it does anything...
 
Interesting find out.

Stange is only that you did not find this before.
Didn't you notice that multitouch does not work on that device before testing your sketches?

And it comes up the question what device are you using there?
Normally on a touch screen you can use two fingers to for example zoom an image.
But if the kernel is compiled for single touch this functionality will not be available.
The touchscreen on that kernel would have very limited functionality.


However I want to recommend here to Paul to create a PJRC Wiki where all these findings are published.
In a category "Simulating touch screen with Teensy" there should be a section explaining that Android has two types of kernels.

There is so much information here in the forum.
If someone searches for a special problem he must read hundreds of forum posts to perhaps find an answer.
Most of the forum posts are old and irrelevant.
Also there are wrong answers in the forum.
And there are posts that are several pages long. It is a waste of time reading that all.
If there was good wiki the Teensy users would save a ton of hours searching for answers.


This petition has already been made in 2015
https://forum.pjrc.com/threads/28793-PJRC-and-Teensy(s)-need-a-Wiki
Paul has never answered.

It could be a kind of wiki where everybody can write but new postings must be checked by Paul before becoming visible.
For Paul this would not be more work than answering here in the forum.
 
Last edited:
The device is a Khadas VIM (1st gen), 2gig ram etc, before this I used it with a mouse, and then the Russian project with a resistive touch screen, which is single touch again, so I had no idea about the multi-touch and single touch kernels, quite strange that they even compile single touch, since multi will always work with single, and it is not like there is a lack of processing power or anything of that sort, probably just an oversight by the image author.

Anyway, here is a link to a video of the Teensy3.2 with Pauls latest usb code running my sketch, FT5316 7" capacitive touchscreen on Android6 (for some reason Nougat image kept crashing when I tried to make a virtualbox screen recording with it). The sketch is probably quite ugly, it is just something I put together to see if the multi-touch actually works, and it does, 2 finger zoom in/out works, the FT5316 controller has some gesture detection capability, but I didn't investigate it yet, I may come back and and update the thread if I do.

youtube video - https://www.youtube.com/watch?v=oO8hy2NsBtM

attached is the sketch and ft5xx6 libraries, I modified them slightly

Might be a good idea to edit the thread name and add [SOLVED] to the name.

View attachment FT5xx6.cppView attachment FT5xx6.hView attachment FT5206.pdfView attachment teensy32_example_ft5316_touchUSB_multitouch.ino
 
However I want to recommend here to Paul to create a PJRC Wiki where all these findings are published.

I'm skeptical a wiki would work. My view on this has changed over time, but this is my current belief.

Or more specifically, I'm pretty sure in the long term, a best case would look much like the Arduino Playground - a vast disorganized collection of unmaintained and out-of-date pages.

If there was good wiki the Teensy users would save a ton of hours searching for answers.

The same could be said for the main website.

I could write a lengthy message here about long-term plans, but I'm going to keep this short and get back to work on a software project.


This petition has already been made in 2015
https://forum.pjrc.com/threads/28793-PJRC-and-Teensy(s)-need-a-Wiki
Paul has never answered.

Why do you say things like this? Maybe you just love speaking with an accusatory tone?

In any case, I did reply several times on that thread started at #57. That was 2 days plus ~9 hours after the thread started.

At that time, nearly 3 years ago, I was more optimistic about the idea of a wiki. Since then, looking at Arduino's wiki, and people complaining about their wiki's content on github issues, and wikis other sites where the site owner doesn't edit or curate, my feeling about a wiki is much less rosy.


It could be a kind of wiki where everybody can write but new postings must be checked by Paul before becoming visible.
For Paul this would not be more work than answering here in the forum.

It's always easy to volunteer other people to do something. It's easy to claim a task you won't be doing doesn't seem like it'll be much work!

Writing with such a harsh tone isn't the best way to win anyone over.
 
Status
Not open for further replies.
Back
Top