Skip to content

Commit

Permalink
Use runtime metrics / new UI (#75)
Browse files Browse the repository at this point in the history
* internal/static: remove semantic-ui

* internal/static: use bootstrap v5

* internal/static: remove semantic-ui accordion config

* internal/static: create plots dynamically

This is a first step towards a dynamic definition of the plots we
show. For this we add a Plot javascript class, which we call to
create/update plotly plots.

* internal/static: move plotly config/layout options in Plot class

* internal/static: extract plot data dynamically (no heatmap yet)

* internal/static: do not repeat plot name in scatter tooltip

* internal/static: Plot.extractData handle heatmaps

* Cosmetics

* internal/static: another step towards dynamic stats definition

This time we're preparing to tackle the stats part, or -how to
extract specific stats from the raw data and assign them to
specific plots-.

* internal/static: split Plot configuration and creation

This is necessary since the application is going to need to
access (in stats.js) to the plots configuration before the plots
are created.

* internal/static: use dynamic plot definition in stats.js

At this point, the only non-dynamic (as in non-externally configurable)
point is the 'datapath' field. That's for now a function in the plot
definition object where we map our raw stats object to a data point.

Eventually, this won't be necessary since this will all be defined in
Go.

* internal/static: minor plot config adjustements, cosmetics

* internal/static: in stats.js remove dead code and cosmetics

* internal/static: improve ws error messages

* internal/static: simplify plot discovery by storing them in a map

* internal/static: fix reconnect bug: replace plots, don't add new ones

* internal/static: move ui.js stuff into app.js and remove file

* Cosmetics

* internal/static: create plots with no data

This will simplify the mini websocket-based protocol we'll have,
breaking down a single do-it-all message into 2, one for the desired
plots configuration - the 'init' message- and the 'data' message,
that will just contain the data.

* internal/static: big data rework, dynamic shapes

* internal/static: move lastGC time conversion upwards

Upwards, as in: closer to Go, where all data processing will eventually
be performed.

* internal/static: remove Plot.name()

* internal/static: simplify Plot._extractData()

* Generate plot definitions from Go template

* s/ioutil/io

* Add an atomic float64 type

* internal/static: use generated plot definitions

* all: milestone! plots are dynamically defined now!

For now we're still plotting runtime.MemStats though

* Add plot def/values for size classes heatmap

* All: code cleaning after switch to generated plot defs

* internal/static: bump plotly to 2.12.1

* Use runtime/metrics

* all: internal/static: add tippy+popper

* all: fixed/clean heatmaps with custom tooltips

* Clean up definitions, split heap into heap{global,details}

* _example: make goroutines count vary

* internal/static: handle 'bar' plots

* Convert to 'bar' and improve units in 'live objects/bytes'

* More way to specify trace color. Change colors for bar plots

* internal/static: use hovermode: 'x' on all but heatmaps

* internal/static: link to runtime/metrics instead of Memstats

* internal/static: improve plot title style

* Resolve TODO in downsampleBucket

* internal/static: remove refs to hoverinfo

* internal/static: extend container width and place 3 plots per row.

* internal/static: improve heatmap toolip

 - Show heatmap tooltip as a table (css).
 - Cleanup code installing the tooltip callbacks.

* Remove unused settings in plotdef.go (TickFormat and DTick)

* Improve heatmap title and axis

* internal/static: cleanup Plot documentation

* Ensure plot defs are always created

When the Go process stops, the web UI tries to reconnect to the
websocket endpoint periodically. When we start a new Go process
'plotdefs' won't be called since the web UI already has all
files. On the Go side, we initialize the plot definitions when we
establish the wsebsocket connection.

We protect plotdefs with sync.Once to avoid concurrent accesses.

* Remove subplot hover (duplicate with subplot name)

* internal/static: make page larger, up to 3 plots

* internal/static: show buckets intervals in heatmaps tooltips

* internal/static: simplify calling createEventShape

* internal/static: move createHorizontalLines to plot.js

* internal/static: cosmetics

* internal/static: use a grid-template css grid

* internal/static: fix 'bad time formatting' warning startup

* internal/static: cosmetics

* internal/plot: move definitions in internal package

* all: simplify plot def structures

* Remove some useless color types

* Simplify plot definitions

* Refactor: each plot gets its own constructor function

* internal/static: rename file

* internal/plot: s/Definition/Config

* Refactor: declare and use plotdef interface

* internal/static: cosmetics change on plot titles

* all: events are shown as vertical lines, not horizontal

Garbage collection were indeed shown correctly as vertical lines,
but they've always been described as being horizontal. Fix this
once and for all I hope :-p

* Wrap metrics/plots handling in plotList

* internal/plot: move plot-related types there

* internal/plot: pre-allocate sizeClasses slice

* internal/plot: move down-sampling tests there

* internal/plot: downsampleCounts allows for pre-allocated slice

* Fix data race when multiple websocket tries to connect

Protect calls to List.Values with a mutex. Also, to avoid having
to copy the buffers, change List.Values() to
List.WriteValues(w io.Writer). WriteValues now directly writes
values converted to JSON into w, so that it's not necessary to
copy internal buffers anymore.

* internal/plot: move stuff

* internal/plot: move and rename stuff

* internal/plot: move and rename stuff

* internal/plot: merge List and allMetrics

* Move plot.List global variable to top-level package

* Fix testIndex

Before, the plots shown on the web page were statically defined.
In testIndex, we can't look anymore for a string containing a
plot title. Instead, just look for id="plots" which is the id
of the <div> where plots are dynamically inserted.

* Fix testWs

Before, the data sent via websocket was static. Now that the set of
plots is dynamic, it would be overkill to exhaustively check all time
series and heatmaps we send, plus that varies from one go version to
another.

Instead, check that we receive 2 data points which presence are likely
guarantee:
 - a time series: the number of goroutines
 - the 'size classes' heatmap

* _example: rework integration tests, use testscript

* _example: convert default, echo and chi examples to testscript

* _example: remove debug print in work.go

* _example: convert fasthttp to testscript

* _example: convert fiber to testscript

* _example: convert gin to testscript

* _example: convert gorilla to testscript

* _example: convert https to testscript

* _example: convert iris to testscript

* _example: convert middleware to testscript

* _example: convert mux to testscript

* _example: convert options to testscript

* _example/testdata: cosmetics

* ci: disable TestExamples verbosity

* internal/plot: UnixMilli doesn't exist in go1.16

* go mod tidy

* Bump minimal Go version to 1.17

* README.md: details about Go version

* internal/plot: use UnixMilli (go1.17+)

* ci: run go mod tidy before testing examples

* ci: switch to checkout and setup-go action v3

* ci: test examples only on latest go

* _example: fix iris dependabot alert (not statsviz-related)

* _example: change server port on fiber example

* add Test_hijack

* internal/static: add modebar button showing plot info in tooltip

* internal/plot: fill infoText for all plots

* internal/plot+static: fix y axis ticks for heatmaps

* Improve handlers tests

* internal: Go-side timestamp rather than javascript

* internal/plot: add CGO calls plot

* internal/static: fix bad heatmaps yaxis on restarts

* internal/plot: add gc stack size plot

* internal/static: always show full time range on x axis

* internal/static: disable zoom/drag on plots

* internal/static: add font-awesome and bootstrap-toggle

* internal/static: add play/pause, show/hide GC and time range

* internal/static: reorganize static file tree

* internal/static: improve plot title cosmetics

* internal/plot: cosmetics

* internal/plot: add new plot: goroutine scheduling events

* internal: cosmetic ui improvement

* Revert "Add an atomic float64 type"

This reverts commit 9d6be37.

* Update CHANGELOG.md and README.md

* internal/plot: merge list.go in plots.go

* internal/plot: homogenize plot names

* Rework README.md

* Update CHANGELOG.md

* Fix TestRegister

* internal/static: use broom icon instead of trashcan for GC lines

* README.md: improve documentation section

* internal/plot: improve plot info text style

* .github: use codecov-action for coverage

* codecov.yml: make coverage informational

* Update README.md

* Update README.md

* Update CONTRIBUTING.md
  • Loading branch information
arl authored Sep 5, 2022
1 parent 583356d commit 25ced36
Show file tree
Hide file tree
Showing 73 changed files with 2,835 additions and 1,601 deletions.
19 changes: 12 additions & 7 deletions .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
on: [push, pull_request]
on: [push]
name: Coverage
jobs:
coverage:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- uses: actions/setup-go@v3
with:
fetch-depth: 2
- uses: actions/setup-go@v2
go-version: 1.19.x
- run: go test -coverprofile=coverage.txt
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
go-version: 1.16.x
- run: go test -coverprofile=coverage.txt && bash <(curl -s https://codecov.io/bash)
fail_ci_if_error: false
files: coverage.txt
name: codecov-umbrella
verbose: true
16 changes: 12 additions & 4 deletions .github/workflows/tests-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,22 @@ jobs:
test:
strategy:
matrix:
go-version: [1.16.x, 1.17.x]
go-version: [1.17.x, 1.18.x, 1.19.x]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@v3
- uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go-version }}
- name: Tests
run: go test -race ./...

test-examples:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-go@v3
with:
go-version: 1.19.x
- name: Test Examples
run: cd _example && go test -race -v .
run: cd _example && go test -race .
6 changes: 3 additions & 3 deletions .github/workflows/tests-others.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ jobs:
os: [macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@v3
- uses: actions/setup-go@v3
with:
go-version: 1.17.x
go-version: 1.19.x
- name: Tests
run: go test -race ./...
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,17 @@
Unreleased yet
==============
* Switch to runtime/metrics as source, major refactor (#75)
+ New heatmap UI component
+ Dynamic plots definition based on server side generated config
+ Add many new plots (scheduler latency, scheduling events, and more)
+ Add play/pause switch button
+ Add show/hide GC events switch button
+ Add time range selector (1m, 5m, 10m)
* Switch javascript code to ES6 (#65)
* Build and test all examples (#63)

v0.4.1 / 2021-12-12
==============
* Assets are `go:embed`ed, so the minimum go version is now go1.16 (#55)
* Polishing (README, small UI improvements) (#54)
* Small ui improvements: link to go.dev rather than golang.org
Expand Down Expand Up @@ -32,6 +42,7 @@ v0.2.2 / 2020-12-13
* Don't log if we can't upgrade to websocket
* `_example`_example: add chi router (#38)
* `_example`_example: change structure to have one example per directory

v0.2.1 / 2020-10-29
===================

Expand Down
27 changes: 14 additions & 13 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,41 +1,42 @@
Contributing
============

First of all, thank you to consider contributing to this open-source project!
First of all, thank you for considering to contribute to statsviz!

Pull-requests are welcome!


## Contribute to statviz Go library
## Go library

The statsviz Go public API is relatively light so there's not much to do and at the moment
it's unlikely that the API will change. However some new options can be added to
`statsviz.Register` without breaking compatibility.

The statsviz Go API is very thin so there's not much to do and it's unlikely that
the API will change, however some new options can be added to `statsviz.Register`
without breaking compatibility.
That being said, there may be things to improve in the implementation, any
contribution is very welcome!

Big changes should be discussed on the issue tracker prior to start working on
the code.

If you've decided to contribute, thank you so much, please comment on the existing
issue or create one stating you want to tackle it, so we can assign it to you and
reduce the possibility of duplicate work.
issue or create one stating what you want to tackle and why.


## Contribute to the user interface (html/css/javascript)
## User interface (html/css/javascript)

The user interface aims to be simple, light and minimal.

Assets are located in the `internal/static` directory and are embedded with
[`go:embed`](https://pkg.go.dev/embed).


## Contribute by improving documentation
## Documentation

No contribution is too small, improvements to code comments and/or README
are welcome!

Thank you!


## Contribute by adding an example
## Examples

There are many Go libraries to handle HTTP routing.

Expand All @@ -54,4 +55,4 @@ example showing how to register statsviz within library `foobar`:
- the example should compile and run
- when ran, statsviz interface should be accessible at http://localhost:8080/debug/statsviz

Thanks a lot!
Thank you!
175 changes: 92 additions & 83 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,64 +1,45 @@
Statsviz
========
# Statsviz

[![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white&style=round-square)](https://pkg.go.dev/github.com/arl/statsviz)
[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge.svg)](https://github.com/avelino/awesome-go)
[![Latest tag](https://img.shields.io/github/tag/arl/statsviz.svg)](https://github.com/arl/statsviz/tag/)


[![Test Actions Status](https://github.com/arl/statsviz/workflows/Tests-linux/badge.svg)](https://github.com/arl/statsviz/actions)
[![Test Actions Status](https://github.com/arl/statsviz/workflows/Tests-others/badge.svg)](https://github.com/arl/statsviz/actions)
[![codecov](https://codecov.io/gh/arl/statsviz/branch/main/graph/badge.svg)](https://codecov.io/gh/arl/statsviz)

<p align="center">
<img alt="Statsviz Gopger Logo" width="250" src="https://raw.githubusercontent.com/arl/statsviz/readme-docs/logo.png?sanitize=true">
<img alt="Statsviz Gopher Logo" width="160" src="https://raw.githubusercontent.com/arl/statsviz/readme-docs/logo.png?sanitize=true">
<img alt="statsviz ui" width="450" align="right" src="https://github.com/arl/statsviz/raw/readme-docs/window.png">
</p>
<br />

Instant Live Visualization of your Go application runtime statistics Heap, Objects, Goroutines, GC, etc.

- Import `"github.com/arl/statsviz"`
- Register statsviz HTTP handlers
- Start your program
- Open your browser at `http://host:port/debug/statsviz`
- Enjoy...

Visualise Go program runtime metrics data in real time: heap, objects, goroutines, GC pauses, scheduler, etc. in your browser.

How does that work?
-----------------

Statsviz serves 2 HTTP handlers.
## Usage

The first one (by default `/debug/statsviz`) serves an html/js user interface showing
some initially empty plots.
Download the latest version:

When you points your browser to statsviz user interface page, it connects to statsviz
second HTTP handler. This second handler then upgrades the connection to the websocket
protocol and starts a goroutine that periodically calls [runtime.ReadMemStats](https://golang.org/pkg/runtime/#ReadMemStats),
sending the result to the user interface, which inturn, updates the plots.
go get github.com/arl/statsviz@latest

Stats are stored in-browser inside a circular buffer which keep tracks of a predefined number of
datapoints, 60, so one minute-worth of data, by default. You can change the frequency
at which stats are sent by passing [SendFrequency](https://pkg.go.dev/github.com/arl/statsviz@v0.2.1#SendFrequency)
to [Register](https://pkg.go.dev/github.com/arl/statsviz@v0.2.1#Register).


Usage
-----

go get github.com/arl/statsviz

Either `Register` statsviz HTTP handlers with the [http.ServeMux](https://pkg.go.dev/net/http?tab=doc#ServeMux) you're using (preferred method):
Register statsviz endpoint on your server [http.ServeMux](https://pkg.go.dev/net/http?tab=doc#ServeMux) (preferred method):

```go
mux := http.NewServeMux()
statsviz.Register(mux)
```

Or register them with the `http.DefaultServeMux`:
Or register on `http.DefaultServeMux`:

```go
statsviz.RegisterDefault()
```

By default statsviz is served at `/debug/statsviz/`.

If your application is not already running an HTTP server, you need to start
one. Add `"net/http"` and `"log"` to your imports and the following code to your
`main` function:
Expand All @@ -69,88 +50,116 @@ go func() {
}()
```

By default the handled path is `/debug/statsviz/`.

Then open your browser at http://localhost:6060/debug/statsviz/.

Go version
----------

You need at least *go1.16* due to the use of [`go:embed`](https://pkg.go.dev/embed).
However you can still use Statsviz with older Go versions, at least *go1.12*, by locking the module to `statsviz@v0.0.4` in your `go.mod`.
## How does that work?

Statsviz serves 2 HTTP endpoints:

- The first one (`/debug/statsviz`) serves a web page with statsviz
user interface, showing initially empty plots.

- The second HTTP handler (`/debug/statsviz/ws`) listens for a WebSocket
connection that will be initiated by statsviz web page as soon as it's loaded in
your browser.

That's it, now your application sends all [runtime/metrics](https://pkg.go.dev/runtime/metrics)
data points to the web page, once per second.

Data points are stored in-browser in a circular buffer which keep tracks of a
predefined number of datapoints.


## Documentation

go get github.com/arl/statsviz@v0.0.4
### Go API

Documentation
-------------
Check out the API reference on [pkg.go.dev](https://pkg.go.dev/github.com/arl/statsviz#section-documentation).

Check out the [API documentation](https://pkg.go.dev/github.com/arl/statsviz#section-documentation).

### User interface

Plots
-----
The controls at the top of the page act on all plots:

On the plots where it matters, garbage collections are shown as vertical lines.
<img alt="menu" src="https://github.com/arl/statsviz/raw/readme-docs/menu-001.png">

### Heap
<img alt="Heap plot image" src="https://github.com/arl/statsviz/raw/readme-docs/heap.png" width="600">
- the groom icon shows/hides the vertical lines representing garbage collections.
- the time range selector defines the visualized time span.
- the play/pause icon allows to stop plots from being refreshed.

### MSpans / MCaches
<img alt="MSpan/MCache plot image" src="https://github.com/arl/statsviz/raw/readme-docs/mspan-mcache.png" width="600">

### Size classes heatmap
<img alt="Size classes heatmap image" src="https://github.com/arl/statsviz/raw/readme-docs/size-classes.png" width="600">
On top of each plot you'll find 2 icons:

### Objects
<img alt="Objects plot image" src="https://github.com/arl/statsviz/raw/readme-docs/objects.png" width="600">
<img alt="menu" src="https://github.com/arl/statsviz/raw/readme-docs/plot.menu-001.png">

### Goroutines
<img alt="Goroutines plot image" src="https://github.com/arl/statsviz/raw/readme-docs/goroutines.png" width="600">
- the camera icon downloads the plot as a PNG image.
- the info icon shows information about the current plot.

### GC/CPU fraction
<img alt="GC/CPU fraction plot image" src="https://github.com/arl/statsviz/raw/readme-docs/gc-cpu-fraction.png" width="600">

#### Plots

Examples
--------
##### Heap (global)
<img alt="Heap (global) image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/heap-global.png">

Have a look at the [_example](./_example/README.md) directory to see various ways to use Statsviz, such as:
- using `http.DefaultServeMux`
- using your own `http.ServeMux`
##### Heap (details)
<img alt="Heap (details) image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/heap-details.png">

##### Live Objects in Heap
<img alt="Live Objects in Heap image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/live-objects.png">

##### Live Bytes in Heap
<img alt="Live Bytes in Heap image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/live-bytes.png">

##### MSpan/MCache
<img alt="MSpan/MCache image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/mspan-mcache.png">

##### Goroutines
<img alt="Goroutines image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/goroutines.png">

##### Size Classes
<img alt="Size Classes image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/size-classes.png">

##### Stop-the-world Pause Latencies
<img alt="Stop-the-world Pause Latencies image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/gc-pauses.png">

##### Time Goroutines Spend in 'Runnable'
<img alt="Time Goroutines Spend in 'Runnable' image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/runnable-time.png">

##### Starting Size of Goroutines Stacks
<img alt="Time Goroutines Spend in 'Runnable' image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/gc-stack-size.png">

##### Goroutine Scheduling Events
<img alt="Time Goroutines Spend in 'Runnable' image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/sched-events.png">

##### CGO Calls
<img alt="CGO Calls image" src="https://github.com/arl/statsviz/raw/readme-docs/runtime-metrics/cgo.png">


## Examples

Check out the [_example](./_example/README.md) directory to see various ways to use Statsviz, such as:
- use of `http.DefaultServeMux` or your own `http.ServeMux`
- wrap HTTP handler behind a middleware
- register at `/foo/bar` instead of `/debug/statviz`
- register the web page at `/foo/bar` instead of `/debug/statviz`
- use `https://` rather than `http://`
- using with various Go HTTP libraries/frameworks:
- register statsviz handlers with various Go HTTP libraries/frameworks:
- [fasthttp](https://github.com/valyala/fasthttp)
- [gin](https://github.com/gin-gonic/gin)
- and many others thanks to awesome contributors!
- and many others thanks to many awesome contributors!


Contributing
------------
## Contributing

Pull-requests are welcome!
More details in [CONTRIBUTING.md](CONTRIBUTING.md).


Roadmap
-------

- [ ] add stop-the-world duration heatmap
- [ ] increase data retention
- [ ] light/dark mode selector
- [x] plot image export as png
- [ ] save timeseries to disk
- [ ] load from disk previously saved timeseries


Changelog
---------
## Changelog

See [CHANGELOG.md](./CHANGELOG.md).


License
-------
## License

- [MIT License](LICENSE)
See [MIT License](LICENSE)
Loading

0 comments on commit 25ced36

Please sign in to comment.