Status monitor and multiple actions on the same client

[grke]

In issues #450 and #566, I have made burp able to process multiple actions for the same client.
For example, restoring whist backing up.

Actually, before these changes, it was possible to list whilst backing up.

But the status monitor has always assumed non-concurrent actions on individual clients.

This task is to make the json output and json input for the monitor able to show multiple actions on one client.

[grke]

As of #450, the actioning child processes are already giving the main parent process enough information for the main parent process to know what clients and pids are running. And the main parent
process now telling the status server child processes the same information.

To do:

1. Make the status server child processes store multiple counter lists for single clients, partitioned by pid, in its memory structures.
2. Make the status server child processes output suitable JSON to represent this information.
3. Make the status server ncurses client able to input the new JSON.
4. Make the status server ncurses client able to display the new information at least to a minimal degree.
5. Do not merge until Ziirish is happy with the new JSON format.

[grke]

I have a patch that does 1, 2, 3.

Here is an example of the updated JSON format.
There is a new section called 'children', into which the "counters" and "phase" sections have moved.

{
    "clients": [
        {
            "name": "testclient",
            "run_status": "running",
            "protocol": 1,
            "children": [
                {
                    "pid": 3943,
                    "backup": 4,
                    "phase": "backup",
                    "counters": [
                        {
                            "name": "files",
                            "type": "f",
                            "count": 0,
                            "changed": 0,
                            "same": 0,
                            "deleted": 0,
                            "scanned": 16
                        },
                        {
                            "name": "directories",
                            "type": "d",
                            "count": 0,
                            "changed": 0,
                            "same": 0,
                            "deleted": 0,
                            "scanned": 1
                        },
                        {
                            "name": "grand_total",
                            "type": "Z",
                            "count": 0,
                            "changed": 0,
                            "same": 0,
                            "deleted": 0,
                            "scanned": 17
                        },
                        {
                            "name": "bytes_estimated",
                            "type": "G",
                            "count": 420151988
                        },
                        {
                            "name": "time_start",
                            "type": "b",
                            "count": 1495345509
                        },
                        {
                            "name": "time_end",
                            "type": "E",
                            "count": 1495345510
                        }
                    ]
                }
            ],
            "backups": [
                {
                    "number": 3,
                    "timestamp": 1421271819,
                    "flags": [
                        "current",
                        "manifest"
                    ],
                    "logs": {
                        "list": [
                            "backup",
                            "backup_stats"
                        ]
                    }
                },
                {
                    "number": 4,
                    "timestamp": 1421272639,
                    "flags": [
                        "working"
                    ],
                    "logs": {
                        "list": [
                            "backup"
                        ]
                    }
                }
            ]
        },
        {
            "name": "laptop",
            "run_status": "idle",
            "backups": [

            ]
        }
    ]
}

[grke]

You can also enter 'j:peer_version=2.1.8' into 'burp -a m' to get the old format.

[ziirish]

I have added an issue on my side so that I do not forget about this change.

There is no blocking point on my side (except me finding some time to actually work on this change), you can merge this whenever you want it's just a 4-5 lines patch in burp-ui.

The changes should be applied in burpui/misc/backend/burp2.py:623.
After giving a quick look at the current code and the changes in the json it looks like there are a couple more things to check.
The main change is the lack of flags field in the testclient output.
From this piece of code:

         backup = None
         phases = ['working', 'finishing']
         for back in client['backups']:
             if 'flags' in back and any([x in back['flags'] for x in phases]):
                 backup = back
                 break
         # check we found a working backup
         if not backup:
             return ret

It looks like there was a flags: ['working', 'finishing'] prior your change, but I guess I'm happy enough with the "run_status": "running" and the "phase": "backup" fields.

[ziirish]

OK, forget about what I just said, the flags field is still present in the backups struct.
So I maintain, this is just a 4-5 lines patch to apply.

Something like:

        backup = None
        try:
            for child in client['children']:
                if 'phase' in child and child['phase'] == 'backup':
                    backup = child
                     break
        except IndexError:
             # fallback to old JSON output format
             phases = ['working', 'finishing']
             for back in client['backups']:
                 if 'flags' in back and any([x in back['flags'] for x in phases]):
                     backup = back
                     break
         # check we found a working backup
         if not backup:
             return ret

+ another little patch around line 690

But if you could add a type key or something this would be awesome. Something to be able to distinguish backups children from restore one. I'm afraid the phase key is not enough since it may have several values for one kind of job (ie. I suppose it may have the values backup, scan, shuffle etc. for the same job).

[grke]

The possible 'phase' entries are these:

#define CNTR_STATUS_STR_SCANNING        "scanning"
#define CNTR_STATUS_STR_BACKUP          "backup"
#define CNTR_STATUS_STR_MERGING         "merging"
#define CNTR_STATUS_STR_SHUFFLING       "shuffling"
#define CNTR_STATUS_STR_LISTING         "listing"
#define CNTR_STATUS_STR_RESTORING       "restoring"
#define CNTR_STATUS_STR_VERIFYING       "verifying"
#define CNTR_STATUS_STR_DELETING        "deleting"
#define CNTR_STATUS_STR_DIFFING         "diffing"

I will add a field next to "phase" called "action", which converts the above to what you want:

const char *cntr_status_to_action_str(struct cntr *cntr)
{
        switch(cntr->cntr_status)
        {
                case CNTR_STATUS_SCANNING:
                case CNTR_STATUS_BACKUP:
                case CNTR_STATUS_MERGING:
                case CNTR_STATUS_SHUFFLING:
                        return "backup";
                case CNTR_STATUS_LISTING:
                        return "list";
                case CNTR_STATUS_RESTORING:
                        return "restore";
                case CNTR_STATUS_VERIFYING:
                        return "verify";
                case CNTR_STATUS_DELETING:
                        return "delete";
                case CNTR_STATUS_DIFFING:
                        return "diff";
                default:
                        return "unknown";
        }
}

Once this passes my automatic tests, I will push the JSON change to master.

[ziirish]

Awesome, thanks!

[grke]

Now in master.

[ziirish]

The new counters should now be supported by burp-ui based on the action key.
But while reading your commit diff I noticed the action key seems missing from docs/status-monitor.txt

[grke]

OK, it is now added, thanks.