Meeting

Flux dev meeting

The Flux developer community holds public meetings to discuss where Flux is going, to get feedback and to create an opportunity to get to know each other face-to-face and in a more immediate fashion than on Slack.

The next meeting is going to take place:

• When: 2019-05-28 15:00 UTC
• Where: Zoom

Before joining, make sure you install the Zoom client and test Microphone/video settings. If you prefer not to be on camera, or just want to listen in, that's absolutely fine too.

Agenda and notes

Agenda

• Review of actions from last meeting

Minutes

• Attendees:
• Video upload:
• Minutes: <add notes here>

Past meetings

• 2019-04-30
• 2019-03-26
• 2019-02-26
• 2018-11-28

2019-04-30

Agenda

• Updates from Weave Flux meeting (Fons, Daniel, Hidde, Michael) in London
• Review of actions from last meeting
• Helm-op 1.0 status

Minutes

• Attendees: Alfonso Acosta, Daniel Holbach, Hidde Beydals, Michael Bridgen, Stefan Prodan
• Video upload: https://zoom.us/recording/play/CkUY45Pgz10ITj_mUV0jZukJznpBXzJIayb6Be-qzQhqZZuJdmi0LZ55ubqnr2Oj?continueMode=true
• Minutes:
  • Updates from Weave Flux meeting (Fons, Daniel, Hidde, Michael) in London
    • Regular release cycle: Helps with predictability ("when does fix X go in?"), we want feature releases (every 2 months) and patch releases (every 2 weeks). We'll use milestones in Github for this.
    • Helm Operator 1.0: image pattern support and git repository split are still to be done.
    • Flux Architecture changes: Fons has been working on enabling Flux to understand generated manifests. Users want to re-use manifests across different environments (using kustomize, etc). Proposal doc, Pull request. Instructions for testing this are in the PR and people are already giving feedback. Also: kubeyaml use will be phased out.
    • Documentation: we want versioned docs - but don't have a clear dream solution for this yet. We are considering adding Flux tutorials to our Katacodas.
      • ACTION: Daniel to do research, start discussion on mailing list about versioned docs
      • ACTION: Daniel to review tutorials, check in with Ben to see how much work it is.
    • End to end testing: e2e testing is already set up, we want more examples added. Some work is: make it easy to run e2e tests locally. Another problem is CircleCI infrastructure being a bit flaky right now - pay for advanced account?
  • Webhooks in Flux: some discussion about webhooks, bitbucket and fluxcloud.
  • Helm Summit: Kingdon wants to present Helm operator at Helm Summit. Some discussion.
    • ACTION: Stefan to collaborate with Kingdon on this.
  • Review actions from last meeting: all done.

2019-03-26

Agenda

• Review of actions from last meeting
• Helm-op 1.0 status
• Next meeting (DST change, etc?)

Minutes

• Attendees: Alfonso Acosta, Daniel Holbach, Hidde Beydals, Michael Bridgen, Stefan Prodan
• Video upload: https://zoom.us/recording/play/3H-85TOJQYyhnOZ7xiOP2cHRnyhPsqPmWEiYuJNq3EdeF_W0sZyuU4MzCe6Qxop-?continueMode=true
• Minutes:
  • Review of actions from last meeting:
    • Fons to send proposal to mailing list too (configuration generators).
      • Hasn't happened yet. Instead Fons was busy working on proof-on-concept.
      • Draft PR, kustomize support issue, kubecfg & jsonnet issue.
      • ACTION: Fons to send to mailing list.
    • Folks to reply on the mailing list to (Justin's) proposal.
      • Looks like discussion is continuing on the list.
    • Bring up Helm Op 1.0 on mailing list, ask for feedback. (Opportunity to ask people for testing before we release.)
      • ACTION: Daniel to add this to meeting minutes.
      • We want to let people know what we feel needs to happen for the Helm operator to be 1.0 material and get feedback.
    • Fons (and Stefan?) to talk to James to figure out the onboarding of Windows package submission. (Michael reminded everyone of more and more Azure people coming in, so quite some interest in Windows.) Folks to coordinate on Slack (and/or the mailing list).
      • Fons pinged Jim on mail. Try mailing list next.
    • Michael/Stefan/Justin to discuss integration tests on the mailing list or Slack. Stefan to start discussion about using Kind (discussion will be: which tests to use, most scripts for testing should be already there). Nick has written skaffolding for running tests already.
      • Stefan did parallel work to Nick's PR: adding end-to-end test infrastructure. Currently stuck at generating SSH keys, Stefan is busy the next days.
      • Uses CircleCI machine as actual VM, spins up Kubernetes Kind in Docker machine. Installs Tiller, build Flux and image, push to Kind cluster. Then apply Helm install of Flux. Kind uses actual code of PR in question, not what's on quay.io. Flux starts up, spin up git container (alpine image, spin up git server, pulls flux-get-started repo), hook up with Flux, check if all workloads of git repo are deployed in the cluster afterwards.
      • SSH secret is currently hard-coded in YAML. Michael has worked on some prior-art. Can share. Stefan is busy right now, so merge now, then file issue and have somebody fix it. Fons is happy to pick it up.
      • Right now only bootstrap test, but all in all we are quite close to using more test-cases. Some discussion about how to run multiple tests and how to parallelise this (using different namespaces, etc).
      • New idea: build fluxctl and use it to sync.
      • End-to-end tests can be run using Kubernetes Kind locally, you only need 3G of RAM.
      • ACTION: Somebody(?) merge Stefan's PR. Somebody(?) file issue about SSH key. Fons to pick it up.
      • Next: review Nick's PR and see which parts can still be used.
  • Helm operator 1.0 status
    • ACTION: Stefan to do Helm chart release.
    • ACTION: Michael to look into splitting helm-operator in its own repo.
    • Supporting even more patterns to specify images in FluxHelmRelease is the most work right now.
    • Hidde is working on an issue related to changing the repository url. This is helm-op-1.0 material now too.
    • ACTION: To send email about helm-op-1.0 project to mailing list.
  • Meeting time change
    • Europe has a DST change coming up which would make the meeting an hour later. As that's where most of the meeting participators are, let's move to 16:00 UTC instead of 17:00 UTC.
    • ACTION: Daniel to plan/announce next meeting.

2019-02-26

Agenda

• Review of actions from last meeting
• Flux roadmap
• Integration tests, next steps
• Plan next meeting

Minutes

• Attendees: Alfonso Acosta, Daniel Holbach, Hidde Beydals, James Pruitt, Justin Barrick, Michael Bridgen, Stefan Prodan, Tamao Nakahara
• Video upload: https://zoom.us/recording/share/2GOFW8UGk7Lu9ob5luUkp6-z8FgVU3BkksTv_QjI8_qwIumekTziMw
• Minutes:
  • Introductions:
    • Justin Barrick, works at Mesosphere, has been contributing "off and on" to Flux and is maintainer of Flux.
    • Alfonso (Fons) Acosta, recently rejoined Weaveworks, works on Flux, not familiar with all the details yet. Very excited about Weave Flux and its activity and community. The vibe is great too.
  • Review of action items
    • ✅ Daniel/Michael to put together strawman for helm-op-1.0 release. TBD in separate "Flux Roadmap" agenda item.
    • Integrations tests, TBD in separate agenda item.
    • ✅ Set up meeting.
    • ✅ Set up mailing list. Here came a reminder for everyone to subscribe to the flux-dev mailing list. It's not meant to replace Slack, but to allow for longer, more structured conversations.
  • Flux Roadmap
    • Michael made clear that there's no published roadmap, but that there is a list of things which are being worked on that are realistic to land soon.
    • Big ticket items to land soon:
      • Signing and fully verifying commits. First half has landed (Hidde): signing everything (sync tag and commits get signed) is done, verification needs to be done still, is about 25% of the remaining part. Alfonso asked if it's a backwards compatibility problem, Hidde clarified that it's behind a flag which needs to be enabled and will only verify commits which land after enabling the flag. Daniel asked if it was going to be turned on by default at some stage. This will be decided after evaluating the user experience because it requires manual steps.
      • Garbage collection of resources. In a good state, to be landed soon. It brings us much to closer to "Flux reflects the state in git in the cluster".
      • Allowing certain namespaces. We'll be graduating this feature from experimental to GA by extending it to cover everything. (Fons worked on this. A couple more weeks.)
    • Experimental, or not started yet:
      • Readonly mode (sync only, don't write sync-tag).
      • Integrate configuration generators into Flux. Not just flat YAML files, e.g. ksonnet/kustomize users. Fons has been working on a PoC, or rather writing a design spec. This is getting even more relevant as kustomize is likely going to get merged into kubernetes.
        • ACTION: Fons to send proposal to mailing list too.
    • Comments and expected changes/features to land in the next months
      • Justin sent a post to the dev list about writing changes in the cluster back to git. His idea was having two daemons, one which would write changes back to git. It would have to live in Flux git, but Justin wasn't sure how people would feel about it. It would make things easier, more composable and enable interesting CI/CD interactions. It'd be quite a bit of work. Fons asked for general overview and a few use-cases. Justin explained that Flux would have rules and heuristics, which would allow Flux to figure out from Kubernetes which changes to commit to git. Flux does this for images already. This would be more generic. Justin's use-case was a backup workflow using the VolumeSnapshot API. He wanted Flux to write the resources to Git, so if the cluster blows up, things could be easily restored (he has an operator which does the backup already, but the information needed to be made persistent). Fons asked if this was going to apply to every resource in the cluster. Justin said it would depend on the heuristics (e.g. applied by kubectl?, change to an annotation, or through a deployment?). Michael added: some bits of manifests like checkpointing HPA data back to git, so it doesn't always start by 0. It'd be hard though to generally rule out feedback loops, so causality only flows one way. Fons wondered if kubediff could be tied in and there'd be a way to easily reconcile changes. The idea being to provide people with better tools to resolve these differences, e.g. "no, it wasn't accident" vs. "yes, please persist this change". A "fluxctl diff" could show you the differences. Stefan suggested that a "dry-run" option could test this, using the new dry-run API (beta in Kubernetes 1.13).
        • ACTION: Folks to reply on the mailing list to proposal.
      • Helm Operator 1.0 roadmap: shopping list of items on the list, we want comments.
        • Bump API version to v1 (Kubernetes CRD version).
        • Put a hook into Helm Operator, so you can poke it and tell it to look at Git (this is in fluxd already).
        • Extend supported formats of images in Helm charts.
        • Break out Helm operator into its own repository. (We'll at least attempt this.)
        • Feedback and questions:
          • What to do if Helm moves to v3? 1) No Tiller, everything client-side. 2) Move to CRD model. Stefan explained which bits will change (don't talk to Tiller any more, let Helm client bootstrap, query CRD through Kubernetes). It will be an interesting challenge to support v2 and v3 and ensure compatibility.
          • Stefan explained that an interesting change will land in Helm (probably backported to Helm v2): use Docker registries for Helm repositories. Maybe use in Helm Operator 1.1?
        • ACTION: to bring up Helm Op 1.0 on mailing list, ask for feedback. (Opportunity to ask people for testing before we release.)
  • fluxctl in Chocolatey: James worked on getting the Windows binary into Chocolatey (Github issue). Binary build and verification process should be owned the Weaveworks org, right now it's in James's Github (link). It'd be great to open this up to more folks and automate it. Testing packages is a bit hard because of provisioning Windows VM where you can install, uninstall and add more unit tests, e.g. virus-scanning or whatever else. KVM can be used, Hyper-V too. There would be the option to leave the verification step to the Chocolatey community. Stefan asked about Appveyor (CI for .NET). Chocolatey seems to use Vagrant, which does not work on Appveyor. Fons has Appveyor experience too - they seem to provide communities with one VM for tests and have pre-built VMs.
    • ACTION: Fons (and Stefan?) to talk to James to figure out the onboarding of Windows package submission. (Michael reminded everyone of more and more Azure people coming in, so quite some interest in Windows.) Folks to coordinate on Slack (and/or the mailing list).
    • Some discussion about Kubernetes entirely on Windows. Stefan clarified that Azure would deploy Flux on a Linux node.
    • Some discussion about Arm builds, some folks pointed towards drone.io. The question: do we want to add build profiles for Arm, Windows, etc.?
  • Integration tests, next steps: the PR has been sitting around for a bit. We ran into issues because it required Kubernetes in an already containerised environment. There was discussion about Docker-on-Docker, or running minikube in CircleCI. Stefan built end-to-end testing of flagger in CircleCI using Kubernetes Kind, which uses Istio and tons of things. Using just Flux should be loads easier - just use a small git server. Justin uses end-to-end test of flux-operator (and other things) in Drone using localkube, but would go the Kubernetes Kind route today. Stefan recommended CircleCI because we already use it and it has enough memory.
    • ACTION: Michael/Stefan/Justin to discuss this on the mailing list or Slack. Stefan to start discussion about using Kind (discussion will be: which tests to use, most scripts for testing should be already there). Nick has written skaffolding for running tests already.
  • Next meeting: agreement on once per month.
    • ACTION: Daniel to plan next meeting.

2018-11-28

Agenda

This is going to be the first meeting, we'll want to get to know each other in our community and get feedback. If you're interested in joining the community, this will be a good place to get involved.

1. Quick introductions (who are you, how do you use Flux)
   If you prefer not to introduce yourself, or just listen in, that's totally fine!
2. State of Flux (Michael)
3. Future plans:
   1. Helm Operator 1.0
   2. Themed releases
   3. Integration test machinery
4. Feedback and questions
5. Next meeting

Minutes

• Attendees: Brant Bobby, Daniel Holbach, Hidde Beydals, Michael Bridgen, Nick Cabatoff, Stefan Prodan
• Video upload: https://zoom.us/recording/play/j3XnvYfXYunpobpanrDZ3F-UarqTye7L0QNk9zS04HjNQR8Aqj5OCSg2D47VzQhq?continueMode=true
• Minutes:
  • Introductions:
    • Michael Bridgen, London, Weaveworks, Lead Dev on Flux, RabbitMQ/VMWare before. Mostly works on Flux.
    • Daniel Holbach, Berlin, Weaveworks, Community Manager, Ubuntu dev communities before. Wants to bring more fun, and a bit more structure into Flux dev community.
    • Nick Cabatoff, used to work at Intelerad, was looking for a CD system, discovered Flux, liked it, decided to get more involved. Moved on to Hashicorp recently.
    • Brant Bobby, works in Canada, looks into transitioning to Kubernetes as a greenfield project. Flux looks like the best option for a CD system there. Is a very enthusiastic user.
    • Hidde Beydals, his story is very similar to Brant's. (Hidde's connection dropped here.)
    • Stefan Prodan, Weaveworks, Developer Experience team. Works on Flux and GitOps.
  • State of Flux (Michael):
    • Flux is not just a Weaveworks project any more. Hidde Beydals, Justin Barrick, Nick Cabatoff joined as maintainers.
    • Flux is used as part of Weaveworks's commercial Weave Cloud product. Rollout progress was added to WC's UI. A PR is in progress to add this to fluxctl release.
    • Another PR that's in the works: Garbage collection for objects which are not part of the Git repo any more.
    • Helm operator: still in beta, but steadily improving. Sees a lot of love through PRs and issues. Let's try to get it to its "1.0" release. It's what multiplied attention to Flux.
    • Supporting syncing multiple Git repos. Very popular request.
    • Support thing like kustomize natively or support composition of CI tooling. Ideas and help welcome!
  • Plans/request/ideas:
    • Helm operator 1.0 - what's missing?
      • Secrets for getting keys for each helm release
      • Some issues open for polish and reliability.
      • Making it less chatty in the logs.
      • Make it follow all of the design spec.
      • No huge drawbacks at this point. Definitely covers the 80%.
      • Let's review open issues and put together a shopping list. Review together. Make it theme of the next set of releases. Ask people to test. Get feedback.
      • ACTION: Daniel/Michael to put together straw-man.
      • Discussion:
        • Hidde: works fine in their setting.
        • Brant: works Helm for external apps, not own apps.
        • Hidde: Use Helm for launching apps with different settings.
        • Brant: Still figuring out how to use Helm for "multi-user support".
        • Somewhat unclear how people organise their environments. Flux needs to stay flexible.
        • Stefan: Flux per namespaces/teams. Clearer structure which responsibilities are with Flux and which are for the helm-operator.
        • Stefan: simple CICD pipeline with Helm releases and Github Actions are possible now.
        • Michael: It'd be great to make Flux more usable with CI pipelines. Waiting for stuff to happen is not easy with Kubernetes.
        • Stefan: use Progress Deadlines? Need a timeout somewhere. Depending on what kind of workload (StatefulSet or not).
        • Michael: let Flux honour deadline, otherwise use Flux setting. Needs design.
        • Stefan: this enables to give feedback to e.g. Github.
        • Michael: previous work on notifications on rollout status (Roli, Elena).
    • Themed releases, ideas
      • User experience: small things like making fluxctl installs easier, crystal-clear log messages.
      • Docs/tutorials: e.g. automated annotations.
    • Integration tests
      • Nick worked on this, some discussion. Let's revive this work. Would be great to get contributors involved with writing tests too. Michael helped with this as well. Raise confidence in not causing regressions.
      • ACTION: Michael/Nick to see what's still missing.
      • Stefan: let's use flux-get-started(?), it covers e.g. all kinds of annotations.
      • Stefan: Damien wrote code to make integration test automations easy. It's used in eksctl.
    • Next meeting
      • ACTION: Daniel: find out on Slack if we need meeting time? Target early next year.
      • Ask for agenda items ahead of time. Try monthly for now. (We compared to Scope's fortnightly meetings.)
      • Focus on design decisions and feedback as opposed to reviewing issues/PRs there.
    • Mailing list
      • Let's use one for long-form design decisions. Slack not really good for asynchronous discussions, not good for planning - some people prefer not to use Slack.
      • ACTION: Michael to set up mailing list.
    • Thanks
      • Thanks to the new maintainers who stepped up!
      • Thanks to everyone who helped out in the past weeks. Flux has got a really great team together!