Big Picture
The Cloud Foundry service broker API is a very easy to implement API that allows to integrate services of almost all kinds into the platform. It does so by focusing on the essential primitives such as offering a catalog and triggering the provisioning of service instances/credentials. The hard part is to actually create the service instances and the even harder part is to keep them operational, i.e. to run the day 2 operations such as backup & restore or updates in an unattended, automated way.
If you have very distinct services, the service broker API will help you tremendously, because it provides you with the necessary “hooks” and goes out of your way for the concrete implementation. On the other hand, if you have services that could be handled similarly, writing a service broker for each of them seems like quite some effort, especially if you start to add the above mentioned day 2 operations.
Services usually aren’t that different and most of them will require some form of backup & restore. Almost all of them will require means to update them. And so it goes on. What you like to have is actually one service broker with harmonized provisioning functionality and day 2 operations for your fleet of services/service instances. That would make it easier for you to offer a compelling service catalog which basically defines the value of your platform.
One could argue that the service broker API could or should be extended, at least by optional operations. However, finding the right abstractions is hard and if the API grows too much, it may no longer fit all services and lead to poor adoption. Plus, you need to understand the domain very well before you can propose such extensions.
So, we started to identify the services that could be handled similarly and what would be needed to do so. We found the cf-containers-broker written by Ferran Rodenas and its BOSH release and started to use and contributed to it (e.g. the Docker/Swarm cluster support, additional service Docker images). We became committers, but eventually departed from that project (mainly because of the limitations we saw in Ruby, parallelism and dealing with multiple streaming connections which is what service brokers usually do) and initiated a separate project which we called the Service Fabrik (German loanword for Factory). The Service Fabrik provisions service instances as Docker and BOSH deployments.
Provide a service provisioning and operations solution:
- Support container- and BOSH-based provisioning (at first)
- Add additional capabilities such as backup & restore and updates (at first)
- Offer a CF command line interface (CLI) plugin for all additional capabilities not natively supported by Cloud Foundry Cloud Controller
- Implement an ops/management UI for the fleet of services/service instances
The features/deliverables of the Service Fabrik are:
- Service broker implementing the Cloud Foundry service broker API and extending it by a remote API offering additional capabilities like user-initiated backup & restore
- Provisioning of service instances to Docker/Swarm utilizing the already existing (16 officially at present) Docker images for the cf-containers-broker/docker-boshrelease
- Provisioning of services to BOSH (“1.0”, planned to switch to BOSH “2.0” feature set, which should be fairly straight-forward as we have something similar to the cloud config)
- Backup & restore supporting OpenStack and AWS (in development; the amount of data is significant, so we need to store the backups in Swift or S3 respectively)
- CF CLI plugin to let the user run backups, list backup sets, or restore them (planned)
- Automated stemcell and release updates (utilizing plainly BOSH)
- Generic service instance dashboards for all services
- Ops/management UI (planned; required for a large fleet of services/service instances; at present, Docker and BOSH help us tremendously to keep everything running, but we see the need for a UI to get a better understanding of the deployment, deal with support requests, extract usage reports and more)
- Automated unit tests
- Developer documentation
Our implementation is currently deployed via BOSH. The catalog is described in the deployment manifest and is generated primarily using spiff, but also ERB. Besides the main catalog data, listing services, their service plans, unique IDs, labels, descriptions, and more, it also contains a section per service plan that describes how to provision a particular service instance. In the case of Docker, a Docker image must be provided and in the case of BOSH, a stemcell and a release must be provided. Whereas the image is basically self-sufficient for Docker, a manifest must be generated for BOSH. That is done by the broker utilizing specially developed manifest templates written in EJS included and encoded on-the-fly into the deployment manifest. This allows to “implement” behavior like generating secure passwords on service creation, but reusing them on updates. The provisioning happens synchronously for Docker and asynchronously for BOSH service instances (in the sense of the service broker API) due to the different nature of the provisioning technologies.
After provisioning, the service instances are ready for consumption/binding and have generic dashboards associated. To utilize the already existing Docker images for the cf-containers-broker/docker-boshrelease without modifications we kept the credential generation logic, but this means there is only one credentials set for a service instance, not per binding. For BOSH service instances we generated binding-unique credential sets through the help of an agent that must be co-deployed with the service instance. The agent must implement the Service Fabrik contract to allow for credential generation/deletion or service-individual backup & restore. Services show different behavior and use different tools, so we didn’t generalize those operations, but basically followed the core principle of the service broker API and provide the service instances merely with hooks to achieve the task.
By updating the Service Fabrik deployment manifest, services/service plans can be added, old ones be deprecated or even deleted (shouldn’t be done if service instances still exist), existing service plans updated, e.g. in the case of BOSH services with new stemcell or release versions. In this case the Service Fabrik will automatically identify service instances of that service plan with an old stemcell or release and trigger a BOSH (re-)deployment, i.e. update. At present this happens when the Service Fabrik is triggered through its remote API, but we plan to allow users to define their own maintenance windows so that they can influence this behavior. Right now, Docker services cannot be updated as we primarily focus on managed BOSH services.
BOSH services can be also updated by the end-user, i.e. the end-user can change the service plan, e.g. to upgrade/downgrade it, be it in terms of functionality, scale or size (should the service only implement certain “t-shirt” sizes which oftentimes is useful for a simplistic billing). The end-user can also keep the service plan and only provide new/different custom parameters that tweak the service behavior, e.g. service specific parameters or even scale it. In the future we plan to improve this with the aforementioned CF CLI plugin.
End-users will be enabled to upgrade service instances through the same means. Compared to an update, upgrade means to actively change the version, e.g. upgrade from PostgreSQL 9.4 to 9.5. Since the Cloud Foundry Cloud Controller meta model has no notion of version, we expose the version in the service plan name, which allows the user to upgrade individually using cf update-service. Part of the catalog data are valid migrations paths between service plans and only those are permitted by the broker. The BOSH release must then be smart enough to support those.
Once we have completed our work on backup & restore, automated updates, and upgrades, we plan to focus our work on an ops/management web UI next.
You will have noticed that adding additional capabilities currently focuses on BOSH services, but we plan to continue our efforts on the container services as well. However, since Docker/Swarm has certain limitations we may need to change the container scheduler first and improve the volume management before we can add more capabilities. This is work that was taken care of by BOSH already as BOSH already has first-class support for many missed features in Docker/Swarm like the volume management. That’s why we are currently focusing on BOSH services, but containers are interesting for managed services as well.
