KurtE
Senior Member+
Sorry, I know this is probably not the best place to ask, this, but was wondering about suggestions for how to reasonably manipulate a large file, to hopefully cut it down to size and maybe make it into a more usable format.
In particular, working on the USB Host code, I often use the Saleae Logic Analyzer to capture USB traffic, for maybe 20-30 seconds. Example trying to figure out the packets that are sent back and forth to a Bluetooth controller to maybe pair with an XBox One controller... I then have the Logic analyzer output the report, which outputs a CSV formatted file which for example might be 50000 lines long... But most of the time, I can manipulate this file down to maybe 100-1500 lines of data that I want to look trhough.
And I find my self doing a bunch of standard search and replace to start reducing this data...
Things like:
a) Remove all SOF files (Start of Frame): Search for:.*,SOF,.*\n replace with nothing...
b) remove all NAK sequences: .*,IN,.*\n.*,NAK,.*\n again replace with nothing
c) Remove all ACK lines: .*,SOF,.*\n
Then I often convert the IN/OUT/SETUP sequences from one two lines to one line: something like:
find: .*,\b(IN|OUT|SETUP)\b,0x[0-9A-F]*,(.*)\n.*DATA[01],,(.*\n)
replaced with: $1,$2,$3
Note: this last one may not have been exact, maybe depends on how many ,,s I need to remove...
This up till now was pretty close how I did all of the postings and the like where I then imported this data into Excel and showed the data in forum posts for Bluetooth and joysticks...
If there was a simple way to do this, it would be great. I currently am using sublimetext search and replace... Thought I would try to make a macro that does this, but their macros don't appear to want to include search/replace stuff.
I have the beginnings of using a regreplace plug in setup to do this... That uses Python script to do the first part of this: Reg_replace_rules...
And then command
And I think it wort of worked, BUT the performance was real bad. That is I waited for several minutes and then went outside with Annie (and Laik) and played for a bit and it was completed when I got back in...
And again this is only a subset....
If I had something that worked well like macros, I would like to expand it to maybe do things like:
Search for lines that state with IN and data area starts with: 0x02 and Insert EV_INQUIRY_RESULT, or if it is 0x17 insert EV_LINK_KEY_REQUEST
That is assume the above had data like: IN,0x01,,017 0x...
Rule like: Search for: (IN,0x01,)(,0x17) replace with: $1EV_INQUIRY_RESULT$2
Likewise for some of the OUT lines, where again if I see something like(OUT,0x01),,(0x01 0x04) insert the word HCI_INQUIRY between the commas...
Suggestions on how to setup such a script or which editor works to do something like this would be appreciated!
Kurt
In particular, working on the USB Host code, I often use the Saleae Logic Analyzer to capture USB traffic, for maybe 20-30 seconds. Example trying to figure out the packets that are sent back and forth to a Bluetooth controller to maybe pair with an XBox One controller... I then have the Logic analyzer output the report, which outputs a CSV formatted file which for example might be 50000 lines long... But most of the time, I can manipulate this file down to maybe 100-1500 lines of data that I want to look trhough.
And I find my self doing a bunch of standard search and replace to start reducing this data...
Things like:
a) Remove all SOF files (Start of Frame): Search for:.*,SOF,.*\n replace with nothing...
b) remove all NAK sequences: .*,IN,.*\n.*,NAK,.*\n again replace with nothing
c) Remove all ACK lines: .*,SOF,.*\n
Then I often convert the IN/OUT/SETUP sequences from one two lines to one line: something like:
find: .*,\b(IN|OUT|SETUP)\b,0x[0-9A-F]*,(.*)\n.*DATA[01],,(.*\n)
replaced with: $1,$2,$3
Note: this last one may not have been exact, maybe depends on how many ,,s I need to remove...
This up till now was pretty close how I did all of the postings and the like where I then imported this data into Excel and showed the data in forum posts for Bluetooth and joysticks...
If there was a simple way to do this, it would be great. I currently am using sublimetext search and replace... Thought I would try to make a macro that does this, but their macros don't appear to want to include search/replace stuff.
I have the beginnings of using a regreplace plug in setup to do this... That uses Python script to do the first part of this: Reg_replace_rules...
Code:
{
"format": "3.0",
"replacements":
{
"remove_usb_packets_ACK":
{
"case": false,
"find": "^.*,ACK,.*\\n",
"greedy": true,
"replace": ""
},
"remove_usb_packets_IN_NAK":
{
"case": false,
"find": "^.*,IN,.*\\n.*,NAK.*\\n",
"greedy": true,
"replace": ""
},
"remove_usb_packets_SOF":
{
"case": false,
"find": "^.*,SOF,.*\\n",
"greedy": true,
"replace": ""
}
}
}
Code:
[
{
"caption": "USB Logic Packet format",
"command": "reg_replace",
"args": {
"replacements": [
"remove_usb_packets_SOF",
"remove_usb_packets_IN_NAK",
"remove_usb_packets_ACK"
]
}
},
]
And again this is only a subset....
If I had something that worked well like macros, I would like to expand it to maybe do things like:
Search for lines that state with IN and data area starts with: 0x02 and Insert EV_INQUIRY_RESULT, or if it is 0x17 insert EV_LINK_KEY_REQUEST
That is assume the above had data like: IN,0x01,,017 0x...
Rule like: Search for: (IN,0x01,)(,0x17) replace with: $1EV_INQUIRY_RESULT$2
Likewise for some of the OUT lines, where again if I see something like(OUT,0x01),,(0x01 0x04) insert the word HCI_INQUIRY between the commas...
Suggestions on how to setup such a script or which editor works to do something like this would be appreciated!
Kurt