Create a public registry interface and separate out HTTP exposition

General context and approch
---------------------------

This is the first part of the long awaited wider refurbishment of
`client_golang/prometheus/...`. After a lot of struggling, I decided
to not go for one breaking big-bang, but cut things into smaller steps
after all, mostly to keep the changes manageable and easy to
review. I'm aiming for having the invasive breaking changes
concentrated in as few steps as possible (ideally one). Some steps
will not be breaking at all, but typically there will be breaking
changes that only affect quite special cases so that 95+% of users
will not be affected. This first step is an example for that, see
details below.

What's happening in this commit?
--------------------------------

This step is about finally creating an exported registry
interface. This could not be done by simply export the existing
internal implementation because the interface would be _way_ to
fat. This commit introduces a very lean `registry.Registry`
interface. Most of the existing functionality that is not part of that
interface is provided by utility functions (in
`registry/util.go`). The top-level function that act on the default
registry are mostly retained and kept in the `prometheus` package, to
minimize breakage and for convenience. The general idea is here that
the basic use case isn't involved with the `registry` package at
all. Only if you are going for one of the advanced use cases, you need
to import the `registry` package.

The default registry is kept in the public variable
`registry.Default`. This follows the example of the http package in
the standard library (cf. `http.DefaultServeMux`,
`http.DefaultClient`).

Another important part in making the registry lean is the extraction
of the HTTP exposition into its own package `promhttp`. Note that the
package is not simply called `http` because that would collide with
the `http` package from the standard library, which will usually be
used in the same source file as the `promhttp` package.

The following issues are fixed by this commit (some solved "on the
fly" now that I was touching the code anyway and it would have been
stupid to port the bugs):

#46
#100
#170
#205

What future changes does this commit enable?
--------------------------------------------

- The separation of the HTTP exposition allows the implementation of
  other exposition methods as known from other Prometheus client
  libraries, e.g. sending the metrics to Graphite.
  Cf. #197

- The public `Registry` interface allows to implement convenience
  tools for testing metric collection. Those tools can inspect the
  collected MetricFamily protobufs and compare them to
  expectation. Also, tests can use their own testing instance of a
  registry.
  Cf. #58

Notable non-goals of this commit
--------------------------------

The following two issues are quite closely connected to the changes in
this commit but the line has been drawn deliberately to address them
in later steps of the refurbishment:

- `InstrumentHandler` has many known problems. The plan is to create a
  saner way to conveniently intrument HTTP handlers in the new
  `promhttp` package and remove the old `InstrumentHandler`
  altogether. To keep breakage low for now, even the default handler
  to expose metrics is still using the old `InstrumentHandler`.
  Cf. #200

- There is work underway to make the whole handling of metric
  descriptors (`Desc`) more intuitive and transparent for the user
  (including an ability for less strict checking,
  cf. #47). That's
  quite invasive from the perspective of the internal code, namely the
  registry. I deliberately kept those changes out of this commit. An
  unfortunate side effect is that the fields of the current `Desc` had
  to be exported for the time being. This will be solved in a cleaner
  way with the upcoming `Desc` changes.

Something that I have played with a lot is "streaming collection",
i.e. allow an implementation of the `Registry` interface that collects
metrics incrementally and serves them while doing so. As it has turned
out, this has many many issues and makes the `Registry` interface very
clunky. Eventually, I made the call that it is unlikely we will really
implement streaming collection, and making the interface more clunky
for something that might not even happen is really a big no-no. Note
that the `Registry` interface only creates the in-memory
representation of the metric family protobufs in one go. The
serializaton onto the wire can still be handled in a streaming
fashion.

What are the breaking changes?
------------------------------

- Signature of functions pushing to Pushgateway (long planned anyway,
  and now that I worked on it anyway, I did it,
  cf. #100).

- `SetMetricFamilyInjectionHook is gone`. A registry with a
  MetricFamily injection hook has to be created now with
  `registry.NewWithInjectionHook`.

- `PanicOnCollectError` is gone. This behavior can now be configured
  when creating a `promhttp.Handler`.

- `EnableCollectChecks` is gone. A registry with those checks can now
  be created with `registry.NewTestRegistry`.

NOTICE

@@ -7,11 +7,6 @@ SoundCloud Ltd. (http://soundcloud.com/).
 
 The following components are included in this product:
 
-goautoneg
-http://bitbucket.org/ww/goautoneg
-Copyright 2011, Open Knowledge Foundation Ltd.
-See README.txt for license details.
-
 perks - a fork of https://github.com/bmizerany/perks
 https://github.com/beorn7/perks
 Copyright 2013-2015 Blake Mizerany, Björn Rabenstein

prometheus/collector.go

@@ -37,16 +37,16 @@ type Collector interface {
 	// executing this method, it must send an invalid descriptor (created
 	// with NewInvalidDesc) to signal the error to the registry.
 	Describe(chan<- *Desc)
-	// Collect is called by Prometheus when collecting metrics. The
-	// implementation sends each collected metric via the provided channel
-	// and returns once the last metric has been sent. The descriptor of
-	// each sent metric is one of those returned by Describe. Returned
-	// metrics that share the same descriptor must differ in their variable
-	// label values. This method may be called concurrently and must
-	// therefore be implemented in a concurrency safe way. Blocking occurs
-	// at the expense of total performance of rendering all registered
-	// metrics. Ideally, Collector implementations support concurrent
-	// readers.
+	// Collect is called by the Prometheus registry when collecting
+	// metrics. The implementation sends each collected metric via the
+	// provided channel and returns once the last metric has been sent. The
+	// descriptor of each sent metric is one of those returned by
+	// Describe. Returned metrics that share the same descriptor must differ
+	// in their variable label values. This method may be called
+	// concurrently and must therefore be implemented in a concurrency safe
+	// way. Blocking occurs at the expense of total performance of rendering
+	// all registered metrics. Ideally, Collector implementations support
+	// concurrent readers.
 	Collect(chan<- Metric)
 }
 

prometheus/doc.go

@@ -84,9 +84,6 @@
 // The Untyped metric behaves like a Gauge, but signals the Prometheus server
 // not to assume anything about its type.
 //
-// Functions to fine-tune how the metric registry works: EnableCollectChecks,
-// PanicOnCollectError, Register, Unregister, SetMetricFamilyInjectionHook.
-//
 // For custom metric collection, there are two entry points: Custom Metric
 // implementations and custom Collector implementations. A Metric is the
 // fundamental unit in the Prometheus data model: a sample at a point in time

prometheus/promhttp/doc.go

@@ -0,0 +1,22 @@
+// Copyright 2016 The Prometheus Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package promhttp provides various tools related to HTTP:
+//
+// - An HTTP handler to server Prometheus metrics via an HTTP endpoint.
+//
+// - Functions to push Prometheus metrics to a Pushgateway.
+//
+// - TODO(beorn7): Convenience functions to instrument HTTP handlers and round
+//   trippers with Prometheus metrics.
+package promhttp

prometheus/promhttp/exposition.go

@@ -0,0 +1,24 @@
+// Copyright 2016 The Prometheus Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package promhttp
+
+import (
+	"net/http"
+
+	"github.com/prometheus/client_golang/prometheus/registry"
+)
+
+func Handler(r registry.Registry) http.Handler {
+	return nil // TODO
+}

prometheus/registry.go

@@ -27,43 +27,18 @@ import (
 	"io"
 	"net/http"
 	"net/url"
-	"os"
 	"sort"
 	"strings"
 	"sync"
 
 	"github.com/golang/protobuf/proto"
 	"github.com/prometheus/common/expfmt"
 
+	"github.com/prometheus/client_golang/prometheus/registry"
 	dto "github.com/prometheus/client_model/go"
 )
 
-var (
-	defRegistry   = newDefaultRegistry()
-	errAlreadyReg = errors.New("duplicate metrics collector registration attempted")
-)
-
-// Constants relevant to the HTTP interface.
 const (
-	// APIVersion is the version of the format of the exported data.  This
-	// will match this library's version, which subscribes to the Semantic
-	// Versioning scheme.
-	APIVersion = "0.0.4"
-
-	// DelimitedTelemetryContentType is the content type set on telemetry
-	// data responses in delimited protobuf format.
-	DelimitedTelemetryContentType = `application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=delimited`
-	// TextTelemetryContentType is the content type set on telemetry data
-	// responses in text format.
-	TextTelemetryContentType = `text/plain; version=` + APIVersion
-	// ProtoTextTelemetryContentType is the content type set on telemetry
-	// data responses in protobuf text format.  (Only used for debugging.)
-	ProtoTextTelemetryContentType = `application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=text`
-	// ProtoCompactTextTelemetryContentType is the content type set on
-	// telemetry data responses in protobuf compact text format.  (Only used
-	// for debugging.)
-	ProtoCompactTextTelemetryContentType = `application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily; encoding=compact-text`
-
 	// Constants for object pools.
 	numBufs           = 4
 	numMetricFamilies = 1000
@@ -100,27 +75,22 @@ func UninstrumentedHandler() http.Handler {
 	return defRegistry
 }
 
-// Register registers a new Collector to be included in metrics collection. It
-// returns an error if the descriptors provided by the Collector are invalid or
-// if they - in combination with descriptors of already registered Collectors -
-// do not fulfill the consistency and uniqueness criteria described in the Desc
-// documentation.
+// Register registers a new Collector to be included in metrics collection with
+// the default registry. It returns an error if the descriptors provided by the
+// Collector are invalid or if they - in combination with descriptors of already
+// registered Collectors - do not fulfill the consistency and uniqueness
+// criteria described in the Desc documentation.
 //
-// Do not register the same Collector multiple times concurrently. (Registering
-// the same Collector twice would result in an error anyway, but on top of that,
-// it is not safe to do so concurrently.)
-func Register(m Collector) error {
-	_, err := defRegistry.Register(m)
-	return err
+// It is in general not safe to register the same Collector multiple
+// times concurrently.
+func Register(c Collector) error {
+	return registry.Default.Register(c)
 }
 
 // MustRegister works like Register but panics where Register would have
 // returned an error.
-func MustRegister(m Collector) {
-	err := Register(m)
-	if err != nil {
-		panic(err)
-	}
+func MustRegister(c Collector) {
+	registry.MustRegister(registry.Default, c)
 }
 
 // RegisterOrGet works like Register but does not return an error if a Collector
@@ -132,77 +102,38 @@ func MustRegister(m Collector) {
 //
 // As for Register, it is still not safe to call RegisterOrGet with the same
 // Collector multiple times concurrently.
-func RegisterOrGet(m Collector) (Collector, error) {
-	return defRegistry.RegisterOrGet(m)
+func RegisterOrGet(c Collector) (Collector, error) {
+	return registry.RegisterOrGet(registry.Default, c)
 }
 
-// MustRegisterOrGet works like Register but panics where RegisterOrGet would
-// have returned an error.
-func MustRegisterOrGet(m Collector) Collector {
-	existing, err := RegisterOrGet(m)
-	if err != nil {
-		panic(err)
-	}
-	return existing
+// MustRegisterOrGet works like RegisterOrGet but panics where RegisterOrGet
+// would have returned an error.
+func MustRegisterOrGet(c Collector) Collector {
+	return registry.MustRegisterOrGet(registry.Default, c)
 }
 
 // Unregister unregisters the Collector that equals the Collector passed in as
-// an argument. (Two Collectors are considered equal if their Describe method
-// yields the same set of descriptors.) The function returns whether a Collector
-// was unregistered.
+// an argument from the default registry. (Two Collectors are considered equal
+// if their Describe method yields the same set of descriptors.) The function
+// returns whether a Collector was unregistered.
+//
+// Note that even after unregistering, it will not be possible to register a new
+// Collector that is inconsistent with the unregistered Collector, e.g. a
+// Collector collecting metrics with the same name but a different help
+// string.
 func Unregister(c Collector) bool {
 	return defRegistry.Unregister(c)
 }
 
-// SetMetricFamilyInjectionHook sets a function that is called whenever metrics
-// are collected. The hook function must be set before metrics collection begins
-// (i.e. call SetMetricFamilyInjectionHook before setting the HTTP handler.) The
-// MetricFamily protobufs returned by the hook function are merged with the
-// metrics collected in the usual way.
-//
-// This is a way to directly inject MetricFamily protobufs managed and owned by
-// the caller. The caller has full responsibility. As no registration of the
-// injected metrics has happened, there is no descriptor to check against, and
-// there are no registration-time checks. If collect-time checks are disabled
-// (see function EnableCollectChecks), no sanity checks are performed on the
-// returned protobufs at all. If collect-checks are enabled, type and uniqueness
-// checks are performed, but no further consistency checks (which would require
-// knowledge of a metric descriptor).
-//
-// Sorting concerns: The caller is responsible for sorting the label pairs in
-// each metric. However, the order of metrics will be sorted by the registry as
-// it is required anyway after merging with the metric families collected
-// conventionally.
-//
-// The function must be callable at any time and concurrently.
-func SetMetricFamilyInjectionHook(hook func() []*dto.MetricFamily) {
-	defRegistry.metricFamilyInjectionHook = hook
-}
-
-// PanicOnCollectError sets the behavior whether a panic is caused upon an error
-// while metrics are collected and served to the HTTP endpoint. By default, an
-// internal server error (status code 500) is served with an error message.
-func PanicOnCollectError(b bool) {
-	defRegistry.panicOnCollectError = b
-}
-
-// EnableCollectChecks enables (or disables) additional consistency checks
-// during metrics collection. These additional checks are not enabled by default
-// because they inflict a performance penalty and the errors they check for can
-// only happen if the used Metric and Collector types have internal programming
-// errors. It can be helpful to enable these checks while working with custom
-// Collectors or Metrics whose correctness is not well established yet.
-func EnableCollectChecks(b bool) {
-	defRegistry.collectChecksEnabled = b
-}
+// TODO port from here on.
 
 // encoder is a function that writes a dto.MetricFamily to an io.Writer in a
 // certain encoding. It returns the number of bytes written and any error
 // encountered.  Note that pbutil.WriteDelimited and pbutil.MetricFamilyToText
 // are encoders.
 type encoder func(io.Writer, *dto.MetricFamily) (int, error)
 
-type registry struct {
+type xxxregistry struct {
 	mtx                       sync.RWMutex
 	collectorsByID            map[uint64]Collector // ID is a hash of the descIDs.
 	descIDs                   map[uint64]struct{}
@@ -673,13 +604,6 @@ func newRegistry() *registry {
 	}
 }
 
-func newDefaultRegistry() *registry {
-	r := newRegistry()
-	r.Register(NewProcessCollector(os.Getpid(), ""))
-	r.Register(NewGoCollector())
-	return r
-}
-
 // decorateWriter wraps a writer to handle gzip compression if requested.  It
 // returns the decorated writer and the appropriate "Content-Encoding" header
 // (which is empty if no compression is enabled).
@@ -694,46 +618,3 @@ func decorateWriter(request *http.Request, writer io.Writer) (io.Writer, string)
 	}
 	return writer, ""
 }
-
-type metricSorter []*dto.Metric
-
-func (s metricSorter) Len() int {
-	return len(s)
-}
-
-func (s metricSorter) Swap(i, j int) {
-	s[i], s[j] = s[j], s[i]
-}
-
-func (s metricSorter) Less(i, j int) bool {
-	if len(s[i].Label) != len(s[j].Label) {
-		// This should not happen. The metrics are
-		// inconsistent. However, we have to deal with the fact, as
-		// people might use custom collectors or metric family injection
-		// to create inconsistent metrics. So let's simply compare the
-		// number of labels in this case. That will still yield
-		// reproducible sorting.
-		return len(s[i].Label) < len(s[j].Label)
-	}
-	for n, lp := range s[i].Label {
-		vi := lp.GetValue()
-		vj := s[j].Label[n].GetValue()
-		if vi != vj {
-			return vi < vj
-		}
-	}
-
-	// We should never arrive here. Multiple metrics with the same
-	// label set in the same scrape will lead to undefined ingestion
-	// behavior. However, as above, we have to provide stable sorting
-	// here, even for inconsistent metrics. So sort equal metrics
-	// by their timestamp, with missing timestamps (implying "now")
-	// coming last.
-	if s[i].TimestampMs == nil {
-		return false
-	}
-	if s[j].TimestampMs == nil {
-		return true
-	}
-	return s[i].GetTimestampMs() < s[j].GetTimestampMs()
-}

prometheus/registry/error.go

@@ -0,0 +1,24 @@
+// Copyright 2016 The Prometheus Authors
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package registry
+
+import "github.com/prometheus/client_golang/prometheus"
+
+type AlreadyRegisteredError struct {
+	ExistingCollector, NewCollector prometheus.Collector
+}
+
+func (err AlreadyRegisteredError) Error() string {
+	return "duplicate metrics collector registration attempted"
+}