@Paul - I simplified the USBSerial claim code where all of them that claim by VID:PID, use the same code to search for the right input and output descriptors, open the pipes... The only difference after that is which packet to send first and setup for which packets will have to be sent after that... Right now doing another quick round of testing, before I check this update in.

One thing I have found during this testing, which I don't think is particular to these changes nor USBSerial in particular. Is I can typically only plug in a Serial device and remove it twice before on the 2nd (sometimes 3rd or 4th) unplug the code hangs. It is hanging in the USBHost::delete_Pipe code. I have instrumented the code with several more Print statements.
Code:
void USBHost::delete_Pipe(Pipe_t *pipe)
{
	println("delete_Pipe ", (uint32_t)pipe, HEX);

	// halt pipe, find and free all Transfer_t

	// EHCI 1.0, 4.8.2 page 72: "Software should first deactivate
	// all active qTDs, wait for the queue head to go inactive"
	//
	// http://www.spinics.net/lists/linux-usb/msg131607.html
	// http://www.spinics.net/lists/linux-usb/msg131936.html
	//
	// In practice it's not feasible to wait for an active QH to become
	// inactive before removing it, for several reasons.  For one, the QH may
	// _never_ become inactive (if the endpoint NAKs indefinitely).  For
	// another, the procedure given in the spec (deactivate the qTDs on the
	// queue) is racy, since the controller can perform a new overlay or
	// writeback at any time.

	bool isasync = (pipe->type == 0 || pipe->type == 2);
	if (isasync) {
		// find the next QH in the async schedule loop
		Pipe_t *next = (Pipe_t *)(pipe->qh.horizontal_link & 0xFFFFFFE0);
		if (next == pipe) {
			// removing the only QH, so just shut down the async schedule
			println("  shut down async schedule");
			USBHS_USBCMD &= ~USBHS_USBCMD_ASE; // disable async schedule
			while (USBHS_USBSTS & USBHS_USBSTS_AS) ; // busy loop wait
			USBHS_ASYNCLISTADDR = 0;
		} else {
			// find the previous QH in the async schedule loop
			println("  remove QH from async schedule");
			Pipe_t *prev = next;
			while (1) {
				Pipe_t *n = (Pipe_t *)(prev->qh.horizontal_link & 0xFFFFFFE0);
				println("    n:", (uint32_t)n, HEX);
				if (n == pipe) break;
				prev = n;
			}
			println("    * Found Pipe in list");
			// if removing the one with H bit, set another
			if (pipe->qh.capabilities[0] & 0x8000) {
				prev->qh.capabilities[0] |= 0x8000; // set H bit
			}
			// link the previous QH, we're no longer in the loop
			prev->qh.horizontal_link = pipe->qh.horizontal_link;
			// do the Async Advance Doorbell handshake to wait to be
			// sure the EHCI no longer references the removed QH
			USBHS_USBCMD |= USBHS_USBCMD_IAA;
			while (!(USBHS_USBSTS & USBHS_USBSTS_AAI)) ; // busy loop wait
			USBHS_USBSTS = USBHS_USBSTS_AAI;
			// TODO: does this write interfere UPI & UAI (bits 18 & 19) ??
		}
		// find & free all the transfers which completed
		println("  Free transfers");
		Transfer_t *t = async_followup_first;
		while (t) {
			print("    * ", (uint32_t)t);
			Transfer_t *next = t->next_followup;
			if (t->pipe == pipe) {
				println(" * remove");
				remove_from_async_followup_list(t);
				free_Transfer(t);
			} else {
				println("");
			}
			t = next;
		}
	} else {
		// remove from the periodic schedule
		for (uint32_t i=0; i < PERIODIC_LIST_SIZE; i++) {
			uint32_t num = periodictable[i];
			if (num & 1) continue;
			Pipe_t *node = (Pipe_t *)(num & 0xFFFFFFE0);
			if (node == pipe) {
				periodictable[i] = pipe->qh.horizontal_link;
				continue;
			}
			Pipe_t *prev = node;
			while (1) {
				num = node->qh.horizontal_link;
				if (num & 1) break;
				node = (Pipe_t *)(num & 0xFFFFFFE0);
				if (node == pipe) {
					prev->qh.horizontal_link = node->qh.horizontal_link;
					break;
				}
				prev = node;
			}
		}
		// TODO: subtract bandwidth from uframe_bandwidth array

		// find & free all the transfers which completed
		Transfer_t *t = periodic_followup_first;
		while (t) {
			Transfer_t *next = t->next_followup;
			if (t->pipe == pipe) {
				remove_from_periodic_followup_list(t);
				free_Transfer(t);
			}
			t = next;
		}
	}
	//
	// TODO: do we need to look at pipe->qh.current ??
	//
	// free all the transfers still attached to the QH
	println("  Free transfers attached to QH");
	Transfer_t *tr = (Transfer_t *)(pipe->qh.next);
	while ((uint32_t)tr & 0xFFFFFFE0) {
		println("    * ", (uint32_t)tr);
		Transfer_t *next = (Transfer_t *)(tr->qtd.next);
		free_Transfer(tr);
		tr = next;
	}
	// hopefully we found everything...
	free_Pipe(pipe);
	println("* Delete Pipe completed");
}
And it is sometimes hanging in the freeing transfers still attached to the QH.

I am wondering if there is a timing issue, like maybe we grab the first one on the list, and start to free it and maybe Interrupt happens that does something to it and the next pointer gets corrupted...

Example Run: Plug in FTDI cable and unplug first time:
Code:
port change: 10001803
    connect
  begin reset
port change: 10001005
  port enabled
  end recovery
new_Device: 12 Mbit/sec
new_Pipe
enumeration:
enumeration:
enumeration:
enumeration:
enumeration:
enumeration:
enumeration:
enumeration:
Config data length = 32
enumeration:
bNumInterfaces = 1
bConfigurationValue = 1
enumeration:
USBHub memory usage = 960
USBHub claim_device this=1FFF2A00
USBHub memory usage = 960
USBHub claim_device this=1FFF3480
HIDParser claim this=1FFF2040
HIDParser claim this=1FFF2520
HIDParser claim this=1FFF3840
USBSerial claim this=1FFF2E40
vid=403, pid=6001, bDeviceClass = 0, bDeviceSubClass = 0, bDeviceProtocol = 0
09 04 00 00 02 FF FF FF 02 07 05 81 02 40 00 00 07 05 02 02 40 00 00 
len = 23
USBSerial, rxep=1, txep=2
new_Pipe
new_Pipe
*** Device USERIAL1 403:6001 - connected ***
  manufacturer: FTDI
  product: TTL232R-3V3
  Serial: FTAKL6VR
control callback (serial) F
control callback (serial) E
control callback (serial) C
control callback (serial) 8
control callback (serial) 0
rx: F8 
�Serial Available
txtimer
  TX data (13) 54 65 73 74 20 6F 66 20 46 54 44 49 0A 
tx1:
rx: 54 65 73 74 20 6F 66 20 46 54 44 49 0A 
Test of FTDI
Serial Available
txtimer
  TX data (12) 53 65 63 6F 6E 64 20 6C 69 6E 65 0A 
tx1:
rx: 53 65 63 6F 6E 64 20 6C 69 6E 65 0A 
Second line
port change: 1C00100A
    disconnect
disconnect_Device:
USBDriver (available_drivers) list: 1FFF2A00 -> 1FFF3480 -> 1FFF2040 -> 1FFF2520 -> 1FFF3840
USBDriver (dev->drivers) list: 1FFF2E40
disconnect driver 1FFF2E40
USBDriver (available_drivers) list: 1FFF2E40 -> 1FFF2A00 -> 1FFF3480 -> 1FFF2040 -> 1FFF2520 -> 1FFF3840
delete_Pipe 1FFF2F20
  remove QH from async schedule
    n:1FFF2EC0
    n:1FFF2F20
    * Found Pipe in list
  Free transfers
    * 536822848 * remove
    * 536822656 * remove
  Free transfers attached to QH
    * 536822656
    * 536822848
    * 536822720
    * 536822784
    * 536817728
    * 536817664
    * 536817600
    * 536817536
    * 536820896
    * 536820832
    * 536820768
    * 536820704
    * 536816000
    * 536815936
    * 536815872
    * 536815808
    * 536814752
    * 536814688
    * 536814624
    * 536814560
    * 536819488
    * 536819424
    * 536819360
    * 536819296
    * 536816800
    * 536816736
    * 536816672
    * 536816608
* Delete Pipe completed
delete_Pipe 1FFF2EC0
  remove QH from async schedule
    n:1FFF2EC0
    * Found Pipe in list
  Free transfers
  Free transfers attached to QH
    * 536817792
* Delete Pipe completed
delete_Pipe 1FFF4300
  shut down async schedule
  Free transfers
  Free transfers attached to QH
    * 536817856
* Delete Pipe completed
removed Device_t from devlist
  disable
*** Device USERIAL1 - disconnected ***
Now I plug it in a 2nd time, Type in some text as I have RX connected to TX line... Then unplug.

Code:
port change: 10001803
    connect
  begin reset
port change: 10001005
  port enabled
  end recovery
new_Device: 12 Mbit/sec
new_Pipe
enumeration:
enumeration:
enumeration:
enumeration:
enumeration:
enumeration:
enumeration:
enumeration:
Config data length = 32
enumeration:
bNumInterfaces = 1
bConfigurationValue = 1
enumeration:
USBSerial claim this=1FFF2E40
vid=403, pid=6001, bDeviceClass = 0, bDeviceSubClass = 0, bDeviceProtocol = 0
09 04 00 00 02 FF FF FF 02 07 05 81 02 40 00 00 07 05 02 02 40 00 00 
len = 23
USBSerial, rxep=1, txep=2
new_Pipe
new_Pipe
*** Device USERIAL1 403:6001 - connected ***
  manufacturer: FTDI
  product: TTL232R-3V3
  Serial: FTAKL6VR
control callback (serial) F
control callback (serial) E
control callback (serial) C
control callback (serial) 8
control callback (serial) 0
Serial Available
txtimer
  TX data (46) 49 20 68 61 76 65 20 61 20 6C 6F 6F 70 62 61 63 6B 20 77 69 72 65 20 61 6E 64 20 74 79 70 65 64 20 69 6E 20 74 68 69 73 20 6C 69 6E 65 0A 
tx1:
rx: 49 20 68 61 76 65 20 61 20 6C 6F 6F 70 62 61 63 6B 20 77 69 72 65 20 61 6E 64 20 74 79 70 65 64 20 69 6E 20 74 68 69 73 20 6C 69 6E 65 0A 
I have a loopback wire and typed in this line
port change: 1C00100A
    disconnect
disconnect_Device:
USBDriver (available_drivers) list: 1FFF2A00 -> 1FFF3480 -> 1FFF2040 -> 1FFF2520 -> 1FFF3840
USBDriver (dev->drivers) list: 1FFF2E40
disconnect driver 1FFF2E40
USBDriver (available_drivers) list: 1FFF2E40 -> 1FFF2A00 -> 1FFF3480 -> 1FFF2040 -> 1FFF2520 -> 1FFF3840
delete_Pipe 1FFF2EC0
  remove QH from async schedule
    n:1FFF4300
    n:1FFF2EC0
    * Found Pipe in list
  Free transfers
    * 536816672 * remove
    * 536816736 * remove
  Free transfers attached to QH
    * 536816736
    * 536816672
    * 536819296
    * 536817792
    * 536819360
    * 536819424
    * 536819488
    * 536814560
    * 536814624
    * 536814688
    * 536814752
    * 536815808
    * 536815872
    * 536815936
    * 536816000
    * 536820704
    * 536820768
    * 536820832
    * 536820896
    * 536817536
    * 536817600
    * 536817664
    * 536817728
    * 536822784
    * 536822720
    * 536822848
    * 536822656
    * 536822656
    * 536822848
    * 536822720
    * 536822784
    * 536817728
    * 536817664
    * 536817600
    * 536817536
    * 536820896
    * 536820832
    * 536820768
    * 536820704
    * 536816000
    * 536815936
    * 536815872
    * 536815808
    * 536814752
    * 536814688
    * 536814624
    * 536814560
    * 536819488
    * 536819424
    * 536819360
    * 536817792
    * 536819296 
... (hangs printing numbers)
Still investigating. But you obviously know a whole lot more about this part of the code!