From d07c5243b2dcf4bd603819f7f3551e6e32b9b6b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jaime=20Rodr=C3=ADguez-Guerra?= Date: Sun, 11 Dec 2022 21:50:23 +0100 Subject: [PATCH 01/32] Initial commit --- blog/2022-12-11-experiment.md | 18 ++++++++++++++++++ blog/authors.yml | 6 ++++++ 2 files changed, 24 insertions(+) create mode 100644 blog/2022-12-11-experiment.md create mode 100644 blog/authors.yml diff --git a/blog/2022-12-11-experiment.md b/blog/2022-12-11-experiment.md new file mode 100644 index 0000000000..eeacff9768 --- /dev/null +++ b/blog/2022-12-11-experiment.md @@ -0,0 +1,18 @@ +--- +slug: docusaurus-experiment +title: Running an experiment with Docusaurus +authors: + - core +tags: [website] +--- + +This prototype website is built with Docusaurus. + + + +We are experimenting with this framework for two reasons: + +- Writing documentation about the conda-forge infrastructure _somewhere_ before we submit it to the official site. +- Seeing if Docusaurus could replace the current conda-forge website. + +This blog post is just a demonstration of how this would look like. diff --git a/blog/authors.yml b/blog/authors.yml new file mode 100644 index 0000000000..a7516fcd8d --- /dev/null +++ b/blog/authors.yml @@ -0,0 +1,6 @@ +core: + name: conda-forge/core + title: The conda-forge core team + url: https://github.com/orgs/conda-forge/teams/core + image_url: https://github.com/conda-forge.png + From 89c55404db80834d3c574e8e8bfdc84c17956f8f Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 27 Jan 2023 14:51:00 +0100 Subject: [PATCH 02/32] Add linters and docs about the life cycle (#1) * add pages about the packaging life cycle * add pre-commit checks * prettier * comment out some stuff * retrigger * use prettier only, with types_or * Some more work on infrastructure components per stage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add some authentication details * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * document some more stuff about feedstocks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * save wip * disable pr autocommit * pre-commit * add WIP marks * change favicon temporarily * a bit more WIP Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- blog/authors.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/blog/authors.yml b/blog/authors.yml index a7516fcd8d..d61b9a5c75 100644 --- a/blog/authors.yml +++ b/blog/authors.yml @@ -3,4 +3,3 @@ core: title: The conda-forge core team url: https://github.com/orgs/conda-forge/teams/core image_url: https://github.com/conda-forge.png - From 15fe8e992ef0d4024f98c46d25315b159b00db67 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:19:52 +0100 Subject: [PATCH 03/32] add blog/2019-12-06-cfep09.md Co-authored-by: cj-wright --- blog/2019-12-06-cfep09.md | 49 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 blog/2019-12-06-cfep09.md diff --git a/blog/2019-12-06-cfep09.md b/blog/2019-12-06-cfep09.md new file mode 100644 index 0000000000..946a0e8d85 --- /dev/null +++ b/blog/2019-12-06-cfep09.md @@ -0,0 +1,49 @@ +--- +authors: + - cj-wright +tags: [autotick-bot] +--- + +# Automatically Deployed ABI Migrations + +Handling application binary interface (ABI) migrations has always been a +hassle for Conda-Forge. Maintaining ABI consistency helps enable the +"just use conda-forge" experience for many of our users, making +certain that numpy's blas is the same as scipy's. As libraries update +their code, the new versions may be ABI incompatible, as function +signatures and other symbols may have changed, leading to the dreaded +`SegmentationFault` and other errors. + + + +Conda-Forge handles this by having a pinning file that tracks all the +currently supported ABIs. These pinned ABIs are then used to build the +downstream packages, making certain that all are consistent. As new +versions of pinned software are released the pins are updated, causing a +migration of the pin, and the rebuilding of all packages which rely on +the pinned package. In the past, this was handled by a change to the +global pinnings and a subsequent migration via the auto-tick bot. While +this worked, there were issues that this created. Firstly, this approach +could cause unsatisfiable build dependencies for new packages, as some +of the new package's dependencies had been compiled with the new pins, +but not all. Secondly, migrations happened in series, if a second pin +was moved while the first was being migrated then the migration could go +wrong as packages which were being rebuilt for the first pin got the +second pin before they were ready. + +Conda-Forge Core has recently approved CFEP-9, a migration policy to fix +these issues. With CFEP-9 pinnings are proposed as small yaml snippets +which contain the new pins. The auto-tick bot then starts migrating the +packages in order, applying the yaml snippet to each package in turn. If +a second pinning change is issued then the bot starts the migration for +that package too, enabling the two migrations to work independently. If +a package needs a change in both pins then the maintainers can choose +the order in which they apply the pins by merging one pin before the +other. + +This approach will yield much greater stability in migrations and will +enable more maintainers to issue migrations. Migrations can be issued by +putting a PR into +[conda-forge/conda-forge-pinning-feedstock](https://github.com/conda-forge/conda-forge-pinning-feedstock), +adding a file to the `migrations` folder, PRs into the auto-tick bot are +not needed anymore. From 1da14141e7838b0d087d24a961d22db3f007f5c8 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:20:51 +0100 Subject: [PATCH 04/32] add blog/2020-02-04-GSoC.md Co-authored-by: ocefpaf --- blog/2020-02-04-GSoC.md | 112 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 blog/2020-02-04-GSoC.md diff --git a/blog/2020-02-04-GSoC.md b/blog/2020-02-04-GSoC.md new file mode 100644 index 0000000000..a3a8ffec6e --- /dev/null +++ b/blog/2020-02-04-GSoC.md @@ -0,0 +1,112 @@ +--- +authors: + - ocefpaf +tags: [autotick-bot, gsoc] +--- + +# Google Summer of Code 2020 improved automatic maintenance of `conda-forge` + +The `conda-forge` "autotick" bot is a crucial part of `conda-forge`'s +infrastructure. It enables automatic maintenance of `conda-forge` +packages by pushing version updates to the underlying software and +enabling large migrations of packages from one dependency to another +(e.g., Python 3.7 to Python 3.8). As `conda-forg` grows in size, with +over 9,000 packages to date, automatic maintenance of the `conda-forge` +ecosystem will become even more important. + + + +We here at `conda-forge` have a large number of potential Google Sumer +of Code projects around maintenance and development of the autotick bot +infrastructure. These projects are high-impact, affecting the entire +`conda-forge` ecosystem. They also cover multiple systems including +databases, `conda`'s CDN provider, continuous integration providers, +and user interactions on GitHub. + +Want to be a part of the team? Great! Take a look at the projects below +and get in touch with us on GitHub! You can check the [GSoC +label]() for a +detailed listing of the issues that need work. + +1. Maintenance and Refactoring + + We have a large backlog of maintenance and refactoring issues that + are great for people with a range of experience from beginners to + true code ninjas. + + - Integration Testing for the Autotick Bot + + > Run true integration testing on a copy of the graph to better + > test code changes and improve our CI process. + > + > **Issue**: + > + > **Experience Level**: advanced + + - Work on the "code hardening" + [Milestone]() + + Address any of the issues in the milestone above related to code + refactoring and cleanup. + + **Issues**: + + **Experience Level**: beginner to advanced + +2. Automated PRs for Globally Pinned Packages + + `conda-forge` maintains a list of globally pinned packages. These + are typically dependencies whose version needs to be the same across + all of `conda-forge` (e.g., the compiler versions or packages like + `HDF5`). While we have infrastructure to run a migration of the + downstream packages from a given pinned package, we do not have + automated infrastructure to propose the migration of the pin itself. + The project here is to add this functionality to our infrastructure. + + **Issue**: + + **Experience Level**: advanced + +3. Check the `conda` CDN for Updated Packages before Issuing PRs in a + Migration + + `conda` relies on a CDN provider to serve the index of available + packages. There is a moderate ~30 minute delay between when a + package is uploaded to `anaconda.org` and when it will appear in the + `conda` index. We currently do not take this delay into account when + issuing PRs in a migration. + + **Issue**: + + **Experience Level**: beginner + +4. Finish Migrations with PR into the `conda-forge` Pinnings File + + Right now, when the migration of say package `ABC` to version `X` + from version `Y` is done, we do not automatically merge the change + in the globally pinned value of `ABC` into our listing of global + pinnings. We should be issuing a PR to the pinnings file once we + have determined that a suitable fraction of the packages effected by + a migration have been properly rebuilt. + + **Issue**: + + **Experience Level**: moderate + +5. Fully Render `conda` Packages in order to Determine Migration + Dependencies + + Determining the dependencies of a given package is actually a + computation expensive task due to the way `conda` recipes are + structured and parametrized through the use of `Jinja2` and + `conda-build-config.yaml` files. Currently, the autotick bot + examines the static metadata in the `meta.yaml` file and not the + fully rendered metadata. For this reason, we sometimes miss + dependencies of a given package that need to be migrated first. + Addressing this issue involves both calling the rendering process + and also scaling that process to the entire set of `conda-forge` + packages. + + **Issue**: + + **Experience Level**: moderate From ec98499bb1ef289dd404de64d5853ca37b12d08c Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:22:31 +0100 Subject: [PATCH 05/32] add blog/2020-03-05-grayskull.md Co-authored-by: marcelotrevisani --- blog/2020-03-05-grayskull.md | 367 +++++++++++++++++++++++++++++++++++ 1 file changed, 367 insertions(+) create mode 100644 blog/2020-03-05-grayskull.md diff --git a/blog/2020-03-05-grayskull.md b/blog/2020-03-05-grayskull.md new file mode 100644 index 0000000000..b77d9498de --- /dev/null +++ b/blog/2020-03-05-grayskull.md @@ -0,0 +1,367 @@ +--- +authors: + - marcelotrevisani +tags: [conda] +--- + +# By the power of Grayskull... I have the Conda recipe! + +_The main goal of the Skeleto_~~n~~_r is to conquer Grayskull._ + +## Introduction + +All jokes aside, the new project +[grayskull](https://github.com/marcelotrevisani/grayskull) was created +with the **intention** of generating better Conda recipes that would +allow to package properly projects available in different channels such +as PyPI, CRAN, Conan, GitHub register, GitHub repositories and so on. On +top of that, Grayskull is also being developed to help +[conda-forge](https://conda-forge.org/) to update recipes. + + + +## Current status + +Currently, Grayskull (version `0.2.1`) is able to generate recipes just +looking for packages on [PyPI](https://pypi.org/), and it is available +on [PyPI](https://pypi.org/project/grayskull/) and +[conda-forge](https://github.com/conda-forge/grayskull-feedstock). The +GitHub repository for this package is: +. + +Before Grayskull, we just had `conda-build skeleton` to generate recipes +for Python packages on PyPI. In all other aspects, the difference of +quality of the generated recipes, and also the time spent to generate +them have a big discrepancy when compared to `conda-build skeleton` and +`grayskull`. Grayskull generates recipes taking in consideration the +platform, Python version available, selectors, compilers (Fortran, C and +C++), packages constrains, license type, license file, and so forth. It +uses metadata available from multiple sources to try to create the best +recipe possible. + +## Installation + +You can install `grayskull` using `pip` or `conda`. `Grayskull` does not +rely on `conda` to run and can generate recipes with minimum +dependencies. + +### With conda + +Grayskull is available on the conda-forge channel. + +```bash +conda install -c conda-forge grayskull +``` + +### With pip + +```bash +pip install grayskull +``` + +## Grayskull vs conda-build skeleton + +There are some differences of recipes generated by `grayskull` and +`conda skeleton`. Taking as example the `pytest` recipe, which has +selectors for platforms, Python version constrains, and has several +package constrains as well. + +### Grayskull (0.2.1) - took 4 seconds to generate the recipe + +```yaml +{% set name = "pytest" %} +{% set version = "5.3.5" %} + +package: + name: {{ name|lower }} + version: {{ version }} + +source: + url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz + sha256: 0d5fe9189a148acc3c3eb2ac8e1ac0742cb7618c084f3d228baaec0c254b318d + +build: + number: 0 + skip: true # [py2k] + entry_points: + - pytest=pytest:main + - py.test=pytest:main + script: {{ PYTHON }} -m pip install . -vv + +requirements: + host: + - pip + - python + - setuptools >=40.0 + - setuptools_scm + run: + - atomicwrites >=1.0 # [win] + - attrs >=17.4.0 + - colorama # [win] + - importlib-metadata >=0.12 # [py<38] + - more-itertools >=4.0.0 + - packaging + - pathlib2 >=2.2.0 # [py<36] + - pluggy <1.0,>=0.12 + - py >=1.5.0 + - python + - wcwidth + +test: + imports: + - pytest + commands: + - pip check + - pytest --help + - py.test --help + requires: + - pip + +about: + home: https://pypi.org/project/pytest/ + summary: 'pytest: simple powerful testing with Python' + dev_url: https://github.com/pytest-dev/pytest + license: MIT + license_file: LICENSE + +extra: + recipe-maintainers: + - marcelotrevisani +``` + +### Skeleton (3.18.11) - took 31 seconds to generate the recipe + +```yaml +{% set name = "pytest" %} +{% set version = "5.3.5" %} + +package: + name: "{{ name|lower }}" + version: "{{ version }}" + +source: + url: "https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz" + sha256: 0d5fe9189a148acc3c3eb2ac8e1ac0742cb7618c084f3d228baaec0c254b318d + +build: + number: 0 + script: "{{ PYTHON }} -m pip install . -vv" + +requirements: + host: + - atomicwrites >=1.0 + - attrs >=17.4.0 + - colorama;sys_platform =="win32" + - importlib-metadata >=0.12 + - more-itertools >=4.0.0 + - packaging + - pathlib2 >=2.2.0 + - pip + - pluggy >=0.12,<1.0 + - py >=1.5.0 + - python + - wcwidth + run: + - atomicwrites >=1.0 + - attrs >=17.4.0 + - colorama;sys_platform =="win32" + - importlib-metadata >=0.12 + - more-itertools >=4.0.0 + - packaging + - pathlib2 >=2.2.0 + - pluggy >=0.12,<1.0 + - py >=1.5.0 + - python + - wcwidth + +about: + home: The package home page + license: MIT + license_family: MIT + license_file: + summary: "pytest: simple powerful testing with Python" + doc_url: + dev_url: + +extra: + recipe-maintainers: + - your-github-id-here +``` + +### Original recipe on `conda-forge` for `pytest 5.3.5` + +```yaml +{% set version = "5.3.5" %} + +package: + name: pytest + version: {{ version }} + +source: + url: https://pypi.io/packages/source/p/pytest/pytest-{{ version }}.tar.gz + sha256: 0d5fe9189a148acc3c3eb2ac8e1ac0742cb7618c084f3d228baaec0c254b318d + +build: + skip: True # [py27] + number: 1 + script: "{{ PYTHON }} setup.py install --single-version-externally-managed --record record.txt" + entry_points: + - py.test = py.test:main + - pytest = py.test:main + +requirements: + host: + - pip + - python + - setuptools >=40.0 + - setuptools_scm + run: + - atomicwrites >=1.0 # [win] + - attrs >=17.4.0 + - colorama # [win] + - importlib_metadata >=0.12 # [py<38] + - more-itertools >=4.0 + - packaging + - pathlib2 >=2.2.0 # [py<36] + - pluggy >=0.12,<1.0 + - py >=1.5.0 + - python + - setuptools >=40.0 + - wcwidth + run_constrained: + # pytest-faulthandler 2 is a dummy package. + # if an older version of fault-handler is installed, it will conflict with pytest >=5. + - pytest-faulthandler >=2 + +test: + commands: + - pytest -h + imports: + - pytest + +about: + home: https://docs.pytest.org/en/latest/ + license: MIT + license_file: LICENSE + summary: 'Simple and powerful testing with Python.' + description: | + The pytest framework makes it easy to write small tests, yet scales to + support complex functional testing for applications and libraries. + doc_url: https://docs.pytest.org/en/latest/ + dev_url: https://github.com/pytest-dev/pytest/ + +extra: + recipe-maintainers: + - flub + - goanpeca + - nicoddemus + - ocefpaf + - mingwandroid +``` + +### Major differences + +| Attribute | Grayskull (0.2.1) | Skeleton (3.18.11) | +| --------- | ----------------- | ------------------ | + **Command** | ✅ grayskull pypi pytest | ✅ conda skeleton pypi pytest | + **Time** | ✅ 4 seconds | ❌ 31 seconds | + **License** | ✅ Added the license file and license type correctly | ❗️ Added just the license type | + **Host Requirements** | ✅ Added correctly all the host requirements | ❌ it didn't add the correctly the host dependencies. It added unnecessary dependencies and it is missing quite a few of them necessary to build the package | + **Run Requirements** | ✅ Missing just setuptools from the host requirements (but this dependency is not defined on pytest package) | ❌ incorrect dependencies added to the project | + **Selectors** | ✅ Skipping correctly Python 2 and added selectors for windows and python versions | ❌ it didn't add any information regarding selectors. Actually conda-build added wrong information which will result in a broken recipe. For example 'sys_platform == win32' was added which is a wrong format for conda recipes | + **Entry points** | ✅ Added all entry points correctly | ❌ No entry points | + **Does it build?** | ✅ YES | ❌ NO | + +In the case of `noarch: python`, Grayskull is smart enough to detect +when the recipe supports it, which is not done by Skeleton. It is +important to highlight that Skeleton does not detect compilers as well. +Nevertheless, Grayskull always try to detect it. + +## Usage Grayskull (0.2.1) + +Project options: + +```bash +$ grayskull --help +usage: grayskull [-h] [--version] {pypi} ... + +Grayskull - Conda recipe generator + +positional arguments: +{pypi} Options to generate PyPI recipes +pypi Generate recipes based on PyPI + +optional arguments: +-h, --help show this help message and exit +--version, -v Print Grayskull version and exit +``` + +```bash +$ grayskull pypi --help +usage: grayskull pypi [-h] [--maintainers MAINTAINERS [MAINTAINERS...]] +[--output OUTPUT] +pypi_packages [pypi_packages ...] + +positional arguments: +pypi_packages Specify the PyPI package name. + +optional arguments: +-h, --help show this help message and exit +--maintainers MAINTAINERS [MAINTAINERS ...], -m MAINTAINERS [MAINTAINERS...] List of maintainers which will be added to the recipe. +--output OUTPUT, -o OUTPUT Path to where the recipe will be created +``` + +To generate the recipe you can just call `grayskull` and pass the +channel (as for now we are just supporting PyPI, it should be pypi) and +the package name. You should also specify an output folder using the +option `--output` or `-o` and it will create the package folder, and the +recipe in there. It is important to note that the user can specify a +list of maintainers which will be added to the recipe using the option +`--maintainers`. + +Example for pytest: + +![Grayskull CLI](https://raw.githubusercontent.com/marcelotrevisani/grayskull/15e4a0317da4e6c2f66a381329682b7e9dc70da0/docs/images/cli_example_grayskull.gif) + +If you need to specify the package version you can do it just puting the +equal sign after the package name and the version just right after that. +Example: + +```bash +grayskull pypi requests=2.21.0 +``` + +or + +```bash +grayskull pypi requests==2.21.0 +``` + +![Grayskull pinned package -requests](https://raw.githubusercontent.com/marcelotrevisani/grayskull/15e4a0317da4e6c2f66a381329682b7e9dc70da0/docs/images/cli_example_grayskull_version.gif) + +If you want to generate multiple recipes just pass a list of packages, +such as: + +```bash +grayskul pypi pytest requests=2.21.0 colorama +``` + +## Future plans + +> - For the next major version (1.0.0) it is planned to add the +> functionality to be able to load the recipe and update just parts +> of it; +> - Generate Conda recipes using CRAN (R) channel (2.0.0); +> - Generate Conda recipes using Conan (C++) channel (3.0.0); + +## Issues + +Any problem, question, suggestions please feel free to open an issue on +the repository: . + +Contributions are very welcome! :) + +------------------------------------------------------------------------ + +This work was possible thanks to the [NumFOCUS](https://numfocus.org/) +Small Development Grant program. From c49e9273e82b6f6536b91bf37ad7f07a7c37f3f3 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:23:06 +0100 Subject: [PATCH 06/32] add blog/2020-03-10-pypy.md Co-authored-by: isuruf --- blog/2020-03-10-pypy.md | 121 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 blog/2020-03-10-pypy.md diff --git a/blog/2020-03-10-pypy.md b/blog/2020-03-10-pypy.md new file mode 100644 index 0000000000..dd2a2f9ea3 --- /dev/null +++ b/blog/2020-03-10-pypy.md @@ -0,0 +1,121 @@ +--- +authors: + - isuruf +tags: [infrastructure] +--- + +# PyPy builds on conda-forge + +conda-forge now supports PyPy3.6 as the python interpreter in a conda +environment + +Supported platforms are, + +> - Linux-x86_64 (glibc 2.12 or newer) +> - OSX-x86_64 (OSX 10.9 or newer) +> - Linux-aarch64 (glibc 2.17 or newer) +> - Linux-ppc64le (glibc 2.17 or newer) + + + +## How to use PyPy + +To use the PyPy builds you can do the following, + +```bash +conda config --set channel_priority strict +conda create -n pypy pypy +conda activate pypy +``` + +If you don't have a conda installation already, you can use +[miniforge-pypy3](https://github.com/conda-forge/miniforge#miniforge-pypy3) +which gives you a conda installation powered by pypy itself. + +However as of the writing of this post, not many conda packages can be +installed into this environment, but noarch packages which do not depend +on the python version nor the interpreter can be installed. For eg, +mpmath is a noarch package without any dependencies. + +```bash +conda install mpmath # succeeds +conda install numpy # fails as of March 10, 2020 +``` + +**UPDATE**: numpy and scipy builds are working as of April 10, 2020. + +All python C extensions needs to be rebuilt for the PyPy ABI. This is +currently on the way and can be tracked at the [status +page](https://conda-forge.org/status). + +## python_abi Package + +As part of adding support for PyPy and to keep the older python builds +working, a python_abi package was added. This defines the abi for the +python package and any non-noarch python packages will have a dependency +on this package. Older python downstream packages like numpy had their +metadata patched to add a CPython ABI. You can ask for a specific python +ABI. + +```bash +conda install "python_abi=*=*_cp27mu" +``` + +If you are using python packages from packages other than defaults, you +will be able to install python extensions built with CPython ABI into +PyPy builds as their metadata have not been patched. Solution in this +case is to hotfix the metadata which is available to only high volume +conda channel or to rebuild those packages with the new python packages +and mark the older ones as broken. + +Using the newer packages will rerender the following requirement to add +a dependency on python_abi. For eg, + +```yaml +requirements: + host: + - python 3.6 + run: + - python +``` + +is rendered as, + +```yaml +requirements: + host: + - python 3.6.10 h9d8adfe_1009_cpython + run: + - python >=3.6,<3.7.0a0 + - python_abi 3.6 *_cp36m +``` + +whereas, + +```yaml +requirements: + host: + - python 3.6 *_73_pypy + run: + - python +``` + +is rendered as, + +```yaml +requirements: + host: + - python 3.6.9 0_73_pypy + run: + - python >=3.6,<3.7.0a0 + - python_abi 3.6 *_pypy36_pp73 +``` + +Note that the PyPy ABI tag has `pp73` at the end which +indicates that the ABI is stable only for PyPy3.6 7.3.x series. + +This opens up the possibility of adding debug builds of python and +building extension packages using the Python Debug ABI. + +Discussion on the PyPy builds can be found in the issue +[conda-forge/conda-forge.github.io#867](https://github.com/conda-forge/conda-forge.github.io/issues/867). From b24c975c6032ab21f13c55797cffac2db93685cf Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:23:34 +0100 Subject: [PATCH 07/32] add blog/2020-07-02-op-risk.md Co-authored-by: cj-wright --- blog/2020-07-02-op-risk.md | 115 +++++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 blog/2020-07-02-op-risk.md diff --git a/blog/2020-07-02-op-risk.md b/blog/2020-07-02-op-risk.md new file mode 100644 index 0000000000..a9eb6b6299 --- /dev/null +++ b/blog/2020-07-02-op-risk.md @@ -0,0 +1,115 @@ +--- +authors: + - cj-wright +tags: [conda-forge] +--- + +# Conda-Forge Operational Risk + +Recently I've been thinking about operational risk (op. risk). +Operational risks arise from failures of processes, for instance a +missing email, or an automated software system not running properly. +Many commercial institutions are interested in minimizing op. risk, +since it is risk that produces no value, as opposed to risks associated +with investing. This is also something I think about in my job at +[Lab49](https://www.lab49.com/), where I'm a software engineering +consultant focusing on financial institutions. I think there is also a +good analogy for Conda-Forge, even though we are not a commercial +outfit. In this case the risk we incur isn't the potential for lost +earnings but frustration for our users and maintainers in the form of +bugs and lackluster user experience. In this post I explore three main +sources of operational risk for Conda-Forge: Automation, Top-Down +Control, and Self-Service Structure. + + + +## A brief conda-forge primer + +Conda-Forge is an ecosystem and community that grew around building +packages for the conda package manager. Conda-Forge uses continuous +integration services to build packages from GitHub repos called +feedstocks. This structure enables teams of contributors to maintain +packages via a pull request based workflow. At time of writing +Conda-Forge has over 10000 feedstocks and ships more than 120 million +packages a month. + +## Self-Service Structure + +Conda-Forge is built around a self-service structure for each stage in a +feedstock's lifecyle. The creation of new feedstocks relies on would be +maintainers to submit PRs to staged-recipes. Although language specific +help teams and staged-recipes reviewers provide some assistance and +oversight, the PR submitter plays the most important role in proposing +the package and shepherding it to acceptance. Once the feedstock is +accepted the maintenance is federated with most upkeep being performed +by the maintainers, who have extensive permissions and control over the +feedstock. If fixes or updates are needed for a package, maintainers and +users are encouraged to open their own pull requests. + +This structure can present a few challenges for minimizing op. risk. The +most important challenge is the disconnect between feedstock maintainers +and users. While most maintainers are package users, most of our users +are not maintainers, and are unlikely to become maintainers. The +disparity between maintainers and users can come from a few sources, +some under our control and others not. For instance we can write better +documentation, lowering the barrier to entry, but we don't have control +over how our user's incentive structures value Conda-Forge +contributions. This produces a gap in representation in the Conda-Forge +organizational structure, where non-maintainer users' issues and +desires are not communicated to maintainers and Core. + +For instance, are we servicing the needs of developers using our +binaries as dependencies to code they are compiling locally. As another +example, are there support gaps for developers and scientists using +Conda-Forge in academic and government laboratories, who might not have +the skills or capacity to fix feedstocks. Our reliance on the public +GitHub platform may prevent some users without access from raising their +concerns. Since these users may be under-represented we don't even know +if we are meeting their needs and how best to help. + +## Top-Down Control + +While the majority of Conda-Forge's permissions structure is federated, +certain important parts are centralized, with the Core developers making +key decisions. Often these decisions are focused on stability of the +ecosystem, for instance what versions of languages to support. +Additionally, maintenance and enhancements to the Conda-Forge +infrastructure are mostly performed by Core developers. + +However, the Core developers are usually experienced feedstock +maintainers, expert conda users, and have bought into the Conda-Forge +ecosystem and mission. This means that decisions can be made without the +perspective of new users or maintainers, or from potential users that +are skeptical of the Conda-Forge approach. + +For instance, decisions about application binary interface pins are +usually made by core, although these changes have impacts on downstream +maintainers. It is possible that most maintainers don't know about what +these pins are, how they are changed and how that affects their +feedstocks. + +## Automation + +Automation has been used to great effect to make Conda-Forge possible. +The various bots and web services enable Conda-Forge's current scale, +providing help and support from running builds, bumping versions, and +checking feedstock quality. However, this automation presents its own +operational risks and magnifies existing operational risks. + +Automation has a tendency to fail when we least expect it and often we +lack the ability to fix it. The January 2018 Travis-CI outage is a great +example of this, where the CI service we were using for macOS builds +experienced reduced capacity and then a complete outage, causing builds +to queue for days. Recently there was a sudden decrease in the number of +parallel builds on Azure causing a similar queue of builds. Automation +can cause issues by enabling users to make decisions without all the +needed information. While many feedstocks have effective smoke tests for +their packages the autotick bot doesn't currently check for new +dependencies, potentially leading to missing or incorrect package +metadata. + +## Conclusion + +Overall Conda-Forge has managed its operational risk well. Most +importantly Conda-Forge's transparent open source nature allows us to +address these issues head on by engaging with the community. From 535f2062624f88b33d661dc36b624d4727ae7228 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:24:02 +0100 Subject: [PATCH 08/32] add blog/2020-07-06-scipy-bof.md Co-authored-by: cj-wright --- blog/2020-07-06-scipy-bof.md | 42 ++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 blog/2020-07-06-scipy-bof.md diff --git a/blog/2020-07-06-scipy-bof.md b/blog/2020-07-06-scipy-bof.md new file mode 100644 index 0000000000..f76d56cd95 --- /dev/null +++ b/blog/2020-07-06-scipy-bof.md @@ -0,0 +1,42 @@ +--- +authors: + - cj-wright +tags: [scipy] +--- + +# Scipy 2020 Packaging BOF + +## Abstract: + +Have some thoughts about conda-forge and how it can be expanded in a way +that is sustainable? Join us in this virtual Birds of a Feather +discussion where we'll discuss maintenance, pain points, opportunities +within conda-forge. Any and all are welcome, and we especially are +seeking new viewpoints and opinions! + + + +## BOF questions: + +### State of Packaging + +- What is the state of conda-forge today? +- What improvements are in the works? + +### New features/ideas + +- What is something that's not in the works that you'd like to see + updated/improved? +- What are ways that folks who are interested but new to the community + can get involved? + +### Maintenance + +- What is the maintenance burden and what pain points does it create? +- What are strategies to mitigate this burden? + +### Questions to the audience + +- What are your pain points when engaging with conda forge? +- What is something new you'd like to see happen? +- What feedback do audience members have? From 38216a55abc8ccb2d02e0400aebbc62afcd80ab8 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:24:26 +0100 Subject: [PATCH 09/32] add blog/2020-07-11-R-4.md Co-authored-by: beckermr Co-authored-by: cj-wright --- blog/2020-07-11-R-4.md | 114 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 blog/2020-07-11-R-4.md diff --git a/blog/2020-07-11-R-4.md b/blog/2020-07-11-R-4.md new file mode 100644 index 0000000000..4c6ffb75ac --- /dev/null +++ b/blog/2020-07-11-R-4.md @@ -0,0 +1,114 @@ +--- +authors: + - cj-wright + - beckermr +tags: [scipy] +--- + +# R 4.0 Migration Retrospective + +While the R 4.0 migration has been functionally complete for quite a +while, the recent migration of `r-java` and its dependents gives a good +opportunity to write a retrospective on the technical issues with +large-scale migrations in `conda-forge` and how we solved them. + +The R 4.0 migration rebuilt every package in `conda-forge` that had +`r-base` as a requirement, including more than 2200 feedstocks. A +migration of this size in `conda-forge` faces several hurdles. First, +since every feedstock is a separate GitHub repository, one needs to +merge more 2200 pull requests (PRs). Second, `conda-forge`'s packages +on `anaconda.org` are behind a CDN (content delivery network). This +service reduces web hosting costs for Anaconda Inc. but introduces an +approximately 30 minute delay from when a package is uploaded to +`anaconda.org` and when it will appear as available using `conda` from +the command line. Thus, even if the dependencies of a package have been +built, we have to wait until they appear on the CDN before we can +successfully issue the next PR and have it build correctly. Finally, the +existing bot and `conda` infrastructure limited the throughput of the +migrations, due in part to the speed of the `conda` solver. + +Given the size of the R 4.0 migration, we took this opportunity to try +out a bunch of new technology to speed up large-scale migrations. The +main enhancements were using GitHub Actions to automerge PRs, using +`mamba` to quickly check for solvability of package environments, and +enabling long-running migration jobs for the autotick bot. All told, the +bulk of the feedstocks for R 4.0 were rebuilt in less than a week, with +many PRs being merged in 30 minutes or less from when they were issued. +These enhancements to the autotick bot and `conda-forge` infrastructure +can be used to enhance future migrations (e.g., Python 3.9) and reduce +maintenance burdens for feedstocks. + +## Automerging conda-forge PRs + +In a typical migration on `conda-forge`, we issue a PR to a feedstock +and then ask the feedstock maintainers to make sure it passes and merge +it. In the case of the R 4.0 migration, the maintainers of R packages on +`conda-forge` use a maintenance team (i.e., `@conda-forge/r`) on the +vast majority of feedstocks. This team is small and so merging over 2000 +PRs by hand is a big undertaking. Thus, with their permission, we added +the `conda-forge` automerge functionality to all R feedstocks that they +maintain. The automerge bot, which relies on GitHub Actions, is able to +automatically merge any PR from the autotick bot that passes the recipe +linter, the continuous integration services, and has the special +`[bot automerge]` slug in the PR title. This feature removed the +bottleneck of waiting for maintainers to merge PRs and reduced the +maintenance burden on the R maintenance team. + +## Checking Solvability with mamba + +While being able to automatically merge PRs removed much of the work of +performing the R 4.0 migration, it relied on the PR building correctly +the first time it was issued. Due to the CDN delays and the build times +of a package's dependencies, the dependencies of a package may not be +immediately available after all of their migration PRs are merged. If +the bot issued the packages migration PR before the dependents are +available, the PR would fail with an unsolvable environment and have to +be restarted manually. This failure would negate any of the benefits of +using automerge in the first place. + +To control for this edge case, we employed the `mamba` package to check +for the solvability of a PR's environments before the PR was issued. +`mamba` is a fast alternative to `conda` that produces solutions for +environments orders of magnitude more quickly. Since, we have to perform +our checks of PR environments many times, an extremely fast solver was +essential for making the code efficient enough to run as part of the +autotick bot. We ended up using mamba to try to install the dependencies +for every variant produced by the feedstock to be migrated. With this +check in place, the autotick bot was able to issue migration PRs that +passed on the first try and were thus automatically merged, many within +30 minutes or less. + +## Improving the Autotick Bot's Efficiency + +Finally, we made several upgrades to the autotick bot infrastructure to +increase the uptime of the bot and its efficiency. First, we moved from +an hourly cron job to a set of chained CI jobs. This change eliminated +downtime between the runs of the bot. Second, we started to refactor the +autotick bot from one monolithic piece of code into a distributed set of +microservices which perform various independent tasks in parallel. These +independent tasks, used for things like checking the statuses of +previously issued PRs, are run separately allowing the bot to spend more +time issuing PRs. Finally, we optimized the internal prioritization of +the PRs to make sure the bot was spending more time on larger migrations +where there is more work to do. More work on the autotick bot +infrastructure, including work done by Vinicius Cerutti as part of the +Google Summer of Code program, will further streamline the bot's +operation. + +Despite some initial hiccups with the bot infrastructure, the migration +ran quite smoothly for an endeavor of its size. The vast majority of +migration PRs were completed within a week from when we started, which +is a first for a migration of this size on `conda-forge`. The largest +issue was solved recently, with the fixing of the `openjdk` recipe and +the removal of `aarch64` and `ppc64le` builds from `r-java`, enabling +the last large piece of the R ecosystem to be updated. + +Looking forward, the improvements we made for the R 4.0 migration seem +broadly applicable to other migration tasks, including the yearly python +minor version bump. These kinds of large-scale migrations are +particularly suitable, since they usually involve few changes to the +feedstock itself and usually fail on CI when a broken package would be +produced. Faster migrations will help to provide the latest features to +downstream users and keep transition times to a minimum, helping to +foster greater stability of the ecosystem and the seamless experience +users have come to expect from `conda-forge`. From f0163cf9b265527700bab3a835a598860547856e Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:24:56 +0100 Subject: [PATCH 10/32] add blog/2020-10-02-versions.md Co-authored-by: cj-wright --- blog/2020-10-02-versions.md | 172 ++++++++++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 blog/2020-10-02-versions.md diff --git a/blog/2020-10-02-versions.md b/blog/2020-10-02-versions.md new file mode 100644 index 0000000000..5d73427de7 --- /dev/null +++ b/blog/2020-10-02-versions.md @@ -0,0 +1,172 @@ +--- +authors: + - cj-wright +tags: [conda-forge] +--- + +# The API Territory and Version Number Map + +tl;dr Depending on specific version numbers of underlying libraries may +be too inaccurate and cause headaches as upstream libraries evolve and +change. A more detailed approach is needed. In this post I outline +current and potential work on a path towards a more complete inspection +of requirements based on APIs and dynamic pinning of libraries. + + + +## What Constitutes a Good Version Number + +Version numbers should constitute a set that has the following +properties + +1. The set must be unbounded +2. The set must be orderable (maybe) + +Of course sets that meet these requirements might not convey a lot of +information about the software they represent other than if two things +are equivalent and their comparative ages. Note that the requirement to +be orderable may not be needed, but is generally useful when considering +the idea of an "upgrade" since it provides a clear delineation between +older and newer packages. In many cases, the structure of the version +number provides additional information. For some projects the version +number includes the date of the release, often using [cal +ver](https://calver.org/). Many projects use [semantic +versioning](https://semver.org/), which attempts to encode information +about the underlying source code's API in the version number. + +## Version Numbers and API Pinning + +One of the most important places where version numbers are specified is +during the pinning of APIs. Source code often requires specific APIs +from the libraries it uses. This requires a pin specifying which +versions of the underlying libraries can be used. The package manager +then uses these pins to make certain a compatible environment is +created. + +However, these pins (or even the lack of pins) produce problems. +Firstly, the pins are a one-time, local statement about the current and +future, global ecosystem of packages. For instance a pin of `scipy` to +the current major version number may not hold up over time, newer +versions of `scipy` may break the API while not changing the major +version number. Similarly the lack of pin for `scipy` could be false as +the API breaks. Even pins that establish firm upper and lower bounds may +be false as new versions of the pinned library restore the missing API. +These issues are particularly problematic for dependency systems that +tie the pins to a particular version of the source code, requiring a new +version to be created to update the pins. Conda-Forge is able to avoid +some of these issues via [repodata +patching](https://github.com/conda-forge/conda-forge-repodata-patches-feedstock), +dynamically updating a package's stated requirements. Overall this +process is fraught, as each package depends on different portions of a +library's API, a version bump that breaks one package may leave others +unscathed. + +## A Potential Path Forward + +All of the above issues are caused by the confusion of [the map for the +territory](https://en.wikipedia.org/wiki/Map%E2%80%93territory_relation). +The map, in this case the version number of a library, can not +accurately represent the territory, the API itself. To fix this issue we +need a more accurate description of the territory. Achieving this will +not be easy, but I think there is an approach that gets close enough to +limit the number of errors. + +We need a programmatic way to check if a particular library, for a +particular version, provides the required API. I think this can be +achieved iteratively, with each step providing additional clarity and +difficulty of implementation. Note that in the steps below, I'm using +python packaging as an example, but I imagine that these steps are +general enough to apply to other languages and ecosystems. + +1. Determine which libraries are requirements of the code, this is + provided by tools like + [depfinder](https://github.com/ericdill/depfinder) and are starting + to be integrated into the Conda-Forge bot systems (although they are + still highly experimental and being worked on). +2. Determine if the a version of the library provides the needed + modules. This could be accomplished by using depfinder to find the + imports and use the mapping provided by + [libcfgraph](https://github.com/regro/libcfgraph/tree/master/import_maps) + between the import names and the versions of packages that ship + those imports. +3. Determine if an imported module provides the symbols being imported. + This would require a listing of all the symbols in a given python + module, including top level scoped variables, function names, class + names, methods, etc. +4. For callables determine if the used call signature matches the + method or function definition. + +The [depfinder](https://github.com/ericdill/depfinder) project has made +significant advances along this path, providing an easy to use tool to +extract accurate import and package requirement data from source code. +Depfinder even has cases to handle imports that are within code blocks +that might make the requirement optional or use the python standard +library. Future work on depfinder, including using more accurate maps +between imports and package names and providing metadata on package +requirements that are collectively exhaustive (for instance imports of +`pyqt4` vs. `pyqt5` in a `try: except:` block), will provide even more +accurate information on requirements. + +At each one of the above stages we can provide significant value to +users, maintainers and source code authors by helping them to keep their +requirements consistent and warning when there are conflicts. +Conda-Forge can update its repodata as new versions of imported +libraries are created, to properly represent if that version is API +compatible with it's downstream consumers. Additionally the tables that +list all the symbols and call signatures can be provided to 3rd party +consumers that may want to patch their own metadata or check if a piece +of source code is self consistent in its requirements. This will also +help with the loosening of pins, creating more solvable environments for +Conda-Forge and other packaging ecosystems. Furthermore, as this tooling +matures and becomes more accurate it can be incorporated into the +Conda-Forge bot systems to automatically update dependencies during +version bumps and repodata patches, helping reduce maintenance burden. + +Tools built from the symbol table can also have impacts far beyond +Conda-Forge. For instance, the symbol tables could allow source code +authors to have a line by line inspection of their code, revealing which +lines force the use of older or newer versions of dependencies. This +could enable large scale migrations of source code with surgical +precision, enabling developers to extract and re-write the few lines of +code preventing the use of a new version of a library. + +## Caveats + +There are some important caveats to this approach that need to be kept +in mind. + +1. All of this work is aimed at understanding the API of a given + library, this approach can not provide insight into the code inside + of the API, or if changes there impact downstream consumers. For + instance, version updates that fix bugs and security flaws in + library code may not change the API at all. From this tooling's + perspective there is no reason to upgrade since the API is not + different. Of course there is a strong reason to upgrade in this + case, since buggy or vulnerable libraries could be a huge headache + and liability for downstream code and should be removed as quickly + as possible. +2. Some features may depend on broader adoption by the community. For + instance, this approach would benefit greatly from python type + hints, since the API could be constrained down to the expected + types. Such type constraints would provide much more accuracy to the + API version range as any changes could be detected. However, type + hints may not be adopted in the python community at a high enough + rate to truly be useful for this application. +3. Source code is fundamentally flexible. There may be knots of code + that even this approach could not cut through, especially as + multiple languages and runtime module loading come into the picture. + My personal hope would be that the code recognizes when these + situations occur, provides its best guess of what is going on, and + provides sufficient metadata to users so that they understand the + decreased accuracy of the results. Fundamentally the tooling can + only provide very educated guesses and context to users, who then + need to go figure out what is actually going on inside the code. + +## Conclusion + +Version number based pins are imprecise representations of API +compatibility. More accurate representations based on source code +inspection would make the Conda-Forge ecosystem more robust and flexible +while reducing maintenance burden. Some of the path to achieving this is +built, and near future steps can be achieved with current tooling and +databases. From 6e3664b3421eddcd90722cd50c9427368d26b495 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:25:13 +0100 Subject: [PATCH 11/32] add blog/2020-10-29-macos-arm64.md Co-authored-by: isuruf --- blog/2020-10-29-macos-arm64.md | 282 +++++++++++++++++++++++++++++++++ 1 file changed, 282 insertions(+) create mode 100644 blog/2020-10-29-macos-arm64.md diff --git a/blog/2020-10-29-macos-arm64.md b/blog/2020-10-29-macos-arm64.md new file mode 100644 index 0000000000..c943a954b9 --- /dev/null +++ b/blog/2020-10-29-macos-arm64.md @@ -0,0 +1,282 @@ +--- +authors: + - isuruf +tags: [conda-forge] +--- + +# macOS ARM builds on conda-forge + +A new platform `osx-arm64` has been added to the build matrix of +conda-forge. `osx-arm64` packages are built to run on upcoming macOS +arm64 processors marketed as `Apple Silicon`. An installer for this +platform can be found +[here](https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh). + + + +This will install a conda environment with python and conda in it. +Installed conda will be able to install packages like `numpy, scipy`. +Currently there are about 100 packages out of 10000 packages pre-built +for this platform. + +All these packages are built on conda-forge's current macOS `x86_64` +infrastructure. In order to do so, we have made lots of changes to the +infrastructure including, +`conda, conda-build, conda-smithy, constructor, conda-forge-ci-setup` to +facilitate cross-compiling which is the process of compiling a package +that will run on a `host` platform (`osx-arm64` in our case), with the +compilation done on a `build` platform (`osx-64` or `linux-64` in our +case). + +`osx-arm64` is the first conda platform that is completely bootstrapped +using conda-build's cross-compiling facility. Previously, when adding a +new platform, conda-build was built with an existing python and pip +environment on the new platform. With cross-compiling, when the +compilers and a sysroot is set up on a different platform, an existing +conda-build installation (on `osx-64` and `linux-64` in this case) will +be able to start building packages right away. + +## Cross-compiling builds for `osx-arm64` + +In order to cross compile packages for `osx-arm64` we need compilers. +So, we first built `clang=11.0.0.rc1` which has support for targetting +`osx-arm64`. We also built `compiler-rt=11.0.0.rc1` as a universal build +support both `osx-64` and `osx-arm64`. + +Linker, archiver, `otool`, `install_name_tool` was built using the +[cctools-port project](https://github.com/tpoechtrager/cctools-port) by +Thomas Pöchtrager. + +One issue we ran into was that the macOS 11.0 Big Sur Beta 7 required +that all executables and shared libraries be ad-hoc signed which is +signing without a verified signature. On suggestion of `cctools-port` +developer we added support to `cctools-port` to sign these executables +using `ldid` which can be used on Linux as well as macOS to sign. + +Using these, the first cross compiled package we built was `libcxx` to +facilitate C++ builds. For the `osx-arm64` sysroot we used the +`MacOSX11.0.sdk` already installed on Azure pipelines and Travis-CI. Due +to licensing issues, we cannot distribute this, but it can be downloaded +from the Apple developer website even on Linux. + +With clang we have a C/C++ compiler, but lack a Fortran compiler. We +used the [GCC fork for +darwin-arm64](https://github.com/iains/gcc-darwin-arm64). First, a cross +compiler (`build == host != target`) was built. Using that compiler, we +built a `cross-native` compiler (`build != host == target`) which gave +use the shared libraries like `libgfortran.dylib`. + +We also added support for cross compiling rust programs to the rust +packages in conda and installing `rust_osx-64` on Linux will give you a +compiler that will build packages for `osx-64`. + +As we haven't done cross-compilation before, many packages needed to be +updated. Most were trivial changes that we automated later on. These +included getting a newer `config.sub` to identify the new autotools +platform `arm64-apple-darwin20.0.0`, adding options to CMake with the +environment variable `CMAKE_ARGS` to correctly set up the toolchain and +recipes were update to use `cmake ${CMAKE_ARGS} ..`. Running tests when +building were also disabled by guarding commands like `make check`, +`make test`, `ctest` with the env variable +`CONDA_BUILD_CROSS_COMPILATION`. + +Cross-compiling python extensions is quite tricky as `distutils` is not +really setup to do this. Thanks to the project +[crossenv](https://github.com/benfogle/crossenv) this is unofficially +supported with a few quirks. With `crossenv`, we can run a python on the +build machine (`osx-64` or `linux-64` in this case) that acts like it is +on `osx-arm64`. `crossenv` monkey-patches a few functions like +`os.uname` and sets up values like `_PYTHON_SYSCONFIG_DATA` to make +python running on `osx-64` or `linux-64` behave like `osx-arm64`. One +issue is that, monkey-patching `sys.platform` doesn't work and +therefore if a python package in it's `setup.py` uses `sys.platform` to +differentiate OSes this will lead to unintended consequences if you are +cross-compiling from `linux-64`. Therefore, we have to use `osx-64` as +our `build` system when cross-compiling for `osx-arm64`. Note that +packages using `sysconfig.get_platform()` will get the correct platform. + +For creating an installer for conda, we needed a standalone conda +executable to bootstrap the conda environment. For other platforms we +relied on `conda-standalone` which is a standalone conda executable +created using `pyinstaller`. Since `pyinstaller` does not support +cross-compile, we decided to use `micromamba` as the bootstrapper and +added features to `micromamba` so that it can function as the +bootstrapper. + +## How to add a `osx-arm64` build to a feedstock + +All the below changes will be done by a bot and the packages the bot +will send PRs to is determined by the list of packages at +[conda-forge-pinning](https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/master/recipe/migrations/osx_arm64.txt) +and their dependences. If you would like to add support, please send a +PR adding the feedstock name to the above list. After that PR is merged, +you can monitor the status at [conda-forge +status-page](https://conda-forge.org/status/#armosxaddition) and if a +particular PR is stalled you can send a PR to the feedstock to fix it. + +Following instructions are for when you want to add support manually. + +Add the following to `conda-forge.yml` (on Linux or OSX), + +```yaml +build_platform: + osx_arm64: osx_64 +test: native_and_emulated +``` + +You can rerender using, + +```bash +conda smithy rerender +``` + +For python packages, add one or more of the following to +`recipe/meta.yaml` as needed, noting that you *must* only add +`numpy`, `cython`, and/or `pybind11` +if they are used in `host:` as well, + +```yaml +requirements: + build: + - python # [build_platform != target_platform] + - cross-python_{{ target_platform }} # [build_platform != target_platform] + - cython # [build_platform != target_platform] + - numpy # [build_platform != target_platform] + - pybind11 # [build_platform != target_platform] +``` + +For autotools package, add the following to `recipe/meta.yaml`, + +```yaml +requirements: + build: + - gnuconfig # [unix] +``` + +and to `recipe/build.sh`, + +```bash +# Get an updated config.sub and config.guess +cp $BUILD_PREFIX/share/gnuconfig/config.* . +``` + +For cmake packages, add the following to `recipe/build.sh`, + +```bash +cmake ${CMAKE_ARGS} .. +``` + +For `meson` packages, add the following to `recipe/build.sh`, + +```bash +meson ${MESON_ARGS} builddir/ +``` + +:::note +Conda automatically creates a [cross build definition +file](https://mesonbuild.com/Cross-compilation.html) when +cross-compiling, and adds the necessary argument to `${MESON_ARGS}` to +point `meson` to that file. `${MESON_ARGS}` is only defined when +cross-compiling, not for normal builds. +::: + +For rust packages, add the following to `recipe/meta.yaml`, + +```bash +requirements: + build: + - {{ compiler('rust') }} +``` + +If there's a line like `make check` in `recipe/build.sh` that cannot be +run when cross-compiling, do the following, + +```bash +if [[ "$CONDA_BUILD_CROSS_COMPILATION" != "1" ]]; then + make check +fi +``` + +After these changes, another rerendering might be required. + +Some useful jinja variables, + +1. `build_platform` - conda subdir for `BUILD_PREFIX`. eg: + `linux-64` +2. `target_platform` - conda subdir for `PREFIX`. eg: `osx-arm64` + +Some useful environment variables, + +1. `build_platform` +2. `target_platform` +3. `CONDA_BUILD_CROSS_COMPILATION` - 1 if cross compiling +4. `CMAKE_ARGS` - arguments to pass to cmake +5. `CC_FOR_BUILD` - C compiler for build platform +6. `CXX_FOR_BUILD` - C++ compiler for build platform +7. `HOST` - a triplet for host passed to autoconf. eg: + `arm64-apple-darwin20.0.0` +8. `BUILD` - a triplet for build passed to autoconf. eg: + `x86_64-conda-linux-gnu` + +Some useful configure options in `conda-forge.yml` + +1. `build_platform` - a dictionary mapping `build` subdir to `host` subdir. eg: + + ```yaml + build_platform: + osx_arm64: osx_64 + linux_ppc64le: linux_64 + linux_aarch64: linux_64 + ``` + +2. `test_on_native_only` - a boolean to turn off testing on cross + compiling. If the tests don't require emulation (for eg: check + that a file exists), then `test_on_native_only: false` will run + the tests even when cross compiling. + +## Building locally + +For building locally add the following in +`$HOME/conda_build_config.yaml`. + +```yaml +SDKROOT: + - /path/to/MacOSX11.0.sdk +``` + +After that, look for the config you want to run in `.ci_support` folder +in the root of the feedstock For eg: `.ci_support/osx_arm64_.yaml`. Then +run, + +```bash +conda build recipe -m .ci_support/osx_arm64_.yaml -c conda-forge -c conda-forge/label/rust_dev +``` + +This should start a new build for `osx-arm64`. + +## Testing packages + +In order to test packages intended to run on future Apple Silicon +hardware, Apple provides a machine called Developer Transition Kit +(DTK). Jonathan Helmus and Eli Rykoff has helped with testing these +packages on DTKs. Thanks to Eli Rykoff, we are now running tests for +these packages as a daily cron job which has led to finding several bugs +in our cross compiling infrastructure and also bugs in our recipes. + +To test cross compiled recipes, transfer the built conda package to the +`host` and run, + +```bash +conda build --test /path/to/package -c conda-forge +``` + +This work would not have been possible without the help of many people +including the upstream maintainers of compiler infrastructure (which +includes conda, conda-build, cctools, tapi, cctools-port, ldid, llvm, +clang, compiler-rt, openmp, libcxx, crossenv, rust, gcc-darwin-arm64), +`conda-forge/help-osx-arm64` team including Matt Becker, Eli Rykoff and +Uwe Korn who sent PRs to fix recipes, `conda-forge/bot` team and also +all the conda-forge maintainers of the 100 feedstocks who reviewed and +fixed PRs. + +Isuru Fernando From 7398deeebf55f02fad14b723ecda704291cd3c10 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:28:41 +0100 Subject: [PATCH 12/32] add blog/2020-11-20-anaconda-tos.md Co-authored-by: beckermr Co-authored-by: cj-wright --- blog/2020-11-20-anaconda-tos.md | 50 +++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 blog/2020-11-20-anaconda-tos.md diff --git a/blog/2020-11-20-anaconda-tos.md b/blog/2020-11-20-anaconda-tos.md new file mode 100644 index 0000000000..d25e8c208a --- /dev/null +++ b/blog/2020-11-20-anaconda-tos.md @@ -0,0 +1,50 @@ +--- +authors: + - core +tags: [conda-forge] +--- + +# Package Distribution and the `anaconda.com` Terms of Service + +Various members of the community have raised questions publicly and +privately about the implications of Anaconda's new Terms of Service +(TOS) on `anaconda.com`. First of all, we understand your concerns. We +would like to explain a bit how `conda-forge` works, how the TOS change +affects us and `conda-forge` users, and what our plans as a community +are for the future. + + + +What makes it non-surprising [that no other free conda package +distribution services have appeared] is that, at the moment, any third +party channel like `conda-forge` is free. The TOS change does not apply +to `conda-forge`, nor to other channels hosted on anaconda.org; the TOS +change in question applies only to the "defaults" channel and other +software hosted on repo.anaconda.com. + +While having alternative hosting is in our plans, we cannot afford the +costs. We are just a community of volunteers. We have experimented with +uploading the `conda-forge` artifacts to GitHub and continue to do so +(see `regro/releases`). We also have put those artifacts behind an +experimental repodata server. One of our core devs (@wolfv) is working +to setup `quetz` with `conda-forge` artifacts as well. + +It is very important to recognize that Anaconda Inc kindly donates use +of their hosting and employee time to us, absorbing all of these costs. +They host about 1.8 TB of our data and serve over 100 million downloads +of artifacts from that data each month. This is a highly significant +donation and `conda-forge` would not exist without it. Anaconda Inc +employees also provide help with maintaining some of the most complex +package recipes in conda-forge. + +Note that Anaconda Inc has also said that part of the revenue from the +TOS change will be donated to OSS projects. (See this blog post: +). +You should be aware that `conda-forge` is a part of NumFOCUS, and so it +stands to benefit from the change in TOS, as do many other OSS projects. + +We absolutely welcome help from the community to move our efforts on +building out more hosting infrastructure for `conda-forge` forward. This +could be anything from spending time developing `quetz` to providing +hosting/mirroring for our data. Please do get in contact with us if +you'd like to help out! From 3e549e549da07374329506623e1872e284f32985 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:29:11 +0100 Subject: [PATCH 13/32] add blog/2020-12-26-year-in-review.md Co-authored-by: beckermr --- blog/2020-12-26-year-in-review.md | 72 +++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 blog/2020-12-26-year-in-review.md diff --git a/blog/2020-12-26-year-in-review.md b/blog/2020-12-26-year-in-review.md new file mode 100644 index 0000000000..6224306ed9 --- /dev/null +++ b/blog/2020-12-26-year-in-review.md @@ -0,0 +1,72 @@ +--- +authors: + - core +tags: [conda-forge] +--- + +# 2020 in Review + +As 2020 winds down, the Core team thought it'd be fun to review some of +the big accomplishments our community has made this year. + +## Strong Growth + +The `conda-forge` community has grown immensely this year. Here are some +numbers to help give you an idea of the scale of our growth. + +- The community has added 3,751 new, unique `conda` packages this + year, along with a corresponding number of new feedstocks. +- For the majority of 2020, the `conda-forge` channel on + `anaconda.org` exceeded 100 million downloads per month. +- In July of 2020, the `conda-forge` channel passed 2 billion total, + all-time downloads. +- We've grown our core developer community, adding seven new members + to the `conda-forge` Core team and at least two members to the + `staged-recipes` team. +- We now have over 2,500 recipe maintainers in the `conda-forge` + GitHub organization. + +## Big New Features + +We've also shipped a ton of big updates to our core infrastructure this +year. These updates include + +- `PyPy` **support**: We added support for `PyPy` 3.6 and now supply + one of the biggest stacks of `PyPy`-enabled packages in the `PyPy` + ecosystem. +- **automerge**: We now support the automatic merging of PRs on + feedstocks using the `automerge` label or through an opt-in setting + in the `conda-forge.yml`. +- `R` **4.0 migration**: This migration was the first one to use our + `automerge` infrastructure at scale. With it, we completed a + complete rebuild/upgrade of the `R` ecosystem in about a week. +- `Python` **updates**: We deprecated `Python` 2.7, completed the + `Python` 3.8 migration, and got about 75% of the way through the + `Python` 3.9 migration. +- **compiler upgrades**: We upgraded our compiler infrastructure to + `GCC` 9 and `clang` 11. +- **CentOS 7 and CentOS 6 EOL**: We shipped an option to enable our + compilers to use the CentOS 7 `sysroot` in preparation for the + CentOS 6 EOL. We hope to complete the move to CentOS 7 early next + year. +- **miniforge**: We built our own standalone, `miniconda`-like + installers. These support a broad range of platforms, including + `osx-arm64` and `linux-aarch64`. +- **standalone Windows stack**: We fully decoupled our Windows recipes + from the `defaults` channel by rebuilding the `msys2` recipes. +- **Apple silicon support**: We added support for Apple silicon with + our `osx-arm64` platform. This platform is our first one to use a + fully cross-compiled infrastructure. +- **CUDA support**: We added support for building CUDA packages on + windows and added CUDA 11.0 support. + +------------------------------------------------------------------------ + +We know that this year has been extremely difficult for so many of our +community members and that the fantastic success of `conda-forge` would +not have been possible without the active participation and support of +our community. **Thank you everyone so much for the work you put into** +`conda-forge` **this year, making it the wonderful, community-led +resource that it is.** + +We wish everyone a happy, healthy, and peaceful new year! From 511dcd0ceb796c87e89b4a89eba4f18367fe4017 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:30:28 +0100 Subject: [PATCH 14/32] add blog/2021-02-02-Outreachy.md Co-authored-by: viniciusdc --- blog/2021-02-02-Outreachy.md | 188 +++++++++++++++++++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 blog/2021-02-02-Outreachy.md diff --git a/blog/2021-02-02-Outreachy.md b/blog/2021-02-02-Outreachy.md new file mode 100644 index 0000000000..3a28d21f22 --- /dev/null +++ b/blog/2021-02-02-Outreachy.md @@ -0,0 +1,188 @@ +--- +authors: + - viniciusdc +tags: [outreachy] +--- + +# Conda-forge Outreachy + +Conda-forge is participating in the upcoming round of +[Outreachy](https://www.outreachy.org/) i.e May 2021 to August 2021. The +goal of this program is to increase participation from under-represented +groups in free and open-source software. Outreachy is organized by +[Software Freedom Conservancy](https://sfconservancy.org/). + + + +## Participant Application Process: + +First, please review the Outreachy Eligibility and Application +Information page to learn more about eligibility for Outreachy. + +## Steps for applicants to conda-forge: + +1. Confirm your eligibility on the Outreachy site +2. Look at the Conda-forge projects available on the Outreachy site, + consider your options, and if you have questions, communicate with the + project mentors. +3. Begin by contributing to the project by looking at our [issues + page](https://github.com/conda-forge/conda-forge.github.io/issues). + As you make contributions, record them on the Outreachy site. +4. Once you have made a few contributions, begin to write your application. + Ask the mentors to review the application before you submit it. + +## Participant Expectations + +You will be working full-time on your project for three months. You will +meet with your mentor(s) frequently and participate in the open-source +development process -- writing code, reviewing code, testing, and so +on. You will be expected to write a blog entry each week. + +## Project Contribution Information + +As part of the application process, all applicants must make at least +one contribution to be accepted as an intern for this project. Only +applicants who make a contribution will be eligible to be accepted as +interns. + +While we don't have one we highly recommend the first-time contributor +to be a conda user and/or submit a package to conda-forge via +[staged-recipes](https://github.com/conda-forge/staged-recipes). That +will ensure the contributor understands the value of what we do and +means that they are willing to participate in our community. + +Applicants can contribute to this project through the [project +repository or contribution page](https://conda-forge.org/#contribute). +The project uses an [issue +tracker](https://github.com/conda-forge/conda-forge.github.io/issues) to +keep information about bugs to fix, project features to implement, +documentation to write, and more. Applicants can look for +newcomer-friendly issues to use for their first contributions by looking +for the following issue tags in the project issue +[tracker](https://github.com/conda-forge/conda-forge.github.io/issues): +Docs, Good first issue + +We here at [conda-forge](https://conda-forge.org/#contribute) have a +large number of potential Outreachy endeavors around documentation, +maintenance, and development. These tasks are high-impact, affecting the +entire conda-forge ecosystem. They also cover multiple systems including +databases, conda's CDN provider, continuous integration providers, and +user interactions on GitHub. + +## How do I work with the conda-forge community? + +Outreachy applicants can get help and feedback from both mentors and +community members. Community members discuss their contributions in a +public chat. Outreachy applicants can often learn from those +discussions. + +Please introduce yourself on the public project chat: + +- Gitter - [Follow this +link](https://gitter.im/conda-forge/conda-forge.github.io) to join this +project's public chat. +- Outreachy mentors will often be in the community public chat. The +project mentor's usernames are: `@viniciusdc`. + +Here are some ready-to-go ways you can get started contributing on your +own. + +- Find an open issue to tackle or report a bug to the issue tracker; +- Don't be afraid to communicate: Ask if you can help write a new + feature or help Automate project setups; +- Improving current tooling and testing features is always welcome. + +As this project main goal is enhancing our current documentation, here +are some preliminary tasks that you can inspect to get ideas: + +- Write and improve the project's documentation; +- Link to duplicate issues, and suggest new issue labels, to keep + things organized; +- Go through open issues and suggest closing old ones; +- Ask clarifying questions on recently opened issues to move the + discussion forward; +- We also have issues regarding the main functionalities of our bot, + in particular the autotick bot. You could find some new information + or ideas for your contributing proposals. + +## Good starter tasks: + +### Small Starter Tasks + +As with most organizations, there are lots of small issues that need +addressing usually related to problems such as bad recipes, old +documentation and others. These will make good first issues to resolve +or "update". This will also be an opportunity to familiarise yourself +with the conda-forge environment. + +### Larger tasks + +There are a few potential larger tasks that can come after a few smaller +task contributions. These are included into our three main bases: + +- [Users](https://conda-forge.org/docs/user/00_intro.html): In this + case, some good starter tasks are mainly checking the actual + contents of conda-forge users documentations, and ideas to better + express its contents. +- [Maintainers](https://conda-forge.org/docs/maintainer/00_intro.html): + There are a bunch of missed topics in this area, some information + have to be updated or rewritten for better understanding. Writing a + complete guide containing the actual steps and standard model for a + package recipe, building process (just a simple discussion) and how + conda-forge bot recognize defective licenses, recipes and packages + in general is highly welcomed. For further understanding of the + general system check [this + link](https://conda-forge.org/docs/maintainer/infrastructure.html). + - It can be funny to say, but lots of helpful ideas and bug + solutions appear on our gitter channel, so if you have time to + write guides about them... it's also an incredible task. +- [And organization](https://conda-forge.org/docs/orga/00_intro.html) + Our environment is changing everyday, because of that a lot of + information is lost in this process or even worse, not documented at + all! which leads to some difficulties inserting new members to + develop and further enhance the current process. + - The related work on this matter is highly welcomed and for a + better grasp of the situation you can start with this + [guideline](https://conda-forge.org/docs/orga/guidelines.html) + and read some of our posts in our + [blog](https://conda-forge.org/blog/blog/) + - Revitalizing ideas/projects for the conda-forge blog are + definitely welcomed; + - Currently we have some interesting projects going on inside our + ecosystem, which in return will need good documentation... Some + of the projects conda-forge is affiliated include the [auto-tick + bot](https://github.com/regro/cf-scripts), + [symbol-exporter](https://github.com/symbol-management/symbol-exporter) + and a new service we are eager to start developing is the + [distributed-bot](https://github.com/regro/cf-scripts/issues/1367). + All of them have a great coverage of subjects and lots of people + to help and give advice about the service structure and + functionalities. + +## Improving the documentation + +You can help improve the documentation as it is version-controlled in +the conda-forge.github.io repository on GitHub. The source text is +stored there in the `src/subdirectory` and is formatted using [Python's +reStructuredText system](https://wiki.python.org/moin/reStructuredText). + +You can propose quick edits directly through the GitHub website, if you +have an account there --- for instance, this +[link](https://github.com/conda-forge/conda-forge.github.io/edit/master/src/user/contributing.rst) +will take you directly to a web-based editor for this section page in +our +[docs](https://conda-forge.org/docs/user/contributing.html#improve-docs). +In general, the file corresponding to each page in the GitHub browser +has a little pencil icon in its top-right that lets you open it up for +editing. + +The more manual process is as follows: + +- Fork the conda-forge.github.io repository to your own GitHub user + account. +- Clone that fork onto your computer. +- Check out a new branch deriving from master to do your work. +- Make and commit your changes. +- Submit a pull request to the main repository proposing your changes. + +Happy editing! From 3f9aec4bce75a20d7f170fab9ee759630fe894f6 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:31:28 +0100 Subject: [PATCH 15/32] add blog/2021-06-16-graykull-step-by-step.md Co-authored-by: ForgottenProgramme --- blog/2021-06-16-graykull-step-by-step.md | 68 ++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 blog/2021-06-16-graykull-step-by-step.md diff --git a/blog/2021-06-16-graykull-step-by-step.md b/blog/2021-06-16-graykull-step-by-step.md new file mode 100644 index 0000000000..2717d12eb0 --- /dev/null +++ b/blog/2021-06-16-graykull-step-by-step.md @@ -0,0 +1,68 @@ +--- +authors: + - name: Mahe + title: Outreachy intern + url: https://github.com/ForgottenProgramme + image_url: https://github.com/ForgottenProgramme.png +tags: [grayskull, outreachy] +--- + +# Contributing Packages To conda-forge Using Grayskull + +When contributing packages to conda-forge, Grayskull can make your life +much easier. Grayskull generates recipes for Python packages hosted on +PyPI. + + + +As the +[introduction](https://github.com/conda-incubator/grayskull#introduction) +for Grayskull reads; "The main goal of this project is to generate +concise recipes for conda-forge." In this tutorial we learn how to +contribute a Python package to the conda-forge channel using Grayskull +to generate the recipe. + +Let us get started. + +1. Install `grayskull` using `conda` through the `conda-forge` channel: + + ``` + $ conda install -c conda-forge grayskull + ``` + +2. Fork and clone the conda-forge [staged-recipes + repository](https://github.com/conda-forge/staged-recipes) from + GitHub. + +3. Checkout a new branch from the `master branch`. + +4. Through CLI, cd inside the 'staged-recipes/recipes' directory. + +5. Call `grayskull` and pass the `pypi` repository, followed by the + name of the package you want to contribute to conda-forge. For + example: `grayskull pypi abc` + + Or you could use `grayskull pypi abc --strict-conda-forge` to remove + some selectors which are not necessary for conda-forge and adapt + recipes to fit better in the conda-forge ecosystem. + + Grayskull will create a folder with the same name as the package (in + this case: 'abc') in the 'recipes' folder of the 'staged-recipes' + directory. This folder will contain the `meta.yaml` file and also + the license file if the package includes a license in the PyPI + distribution. + +6. Go through the generated `meta.yaml` file. For simpler packages, the + generated recipes are nearly perfect, but for some packages you + might need to make certain tweaks. + +7. Commit and push the changes. `git add recipe/abc/meta.yaml` + `git commit -m "add a commit message"` `git push` + +8. Create a PR. + +9. Once the CI is passing, post a comment saying: + `This is ready for review` `@conda-forge-admin, please ping team` + +Once the PR gets merged, your package will be available on the +conda-forge channel. Tada! It's that easy. From 769c5cb64985ca6eff5cfd2f72f524fcdc161ae5 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:31:51 +0100 Subject: [PATCH 16/32] add blog/2021-09-24-travis-security.md Co-authored-by: beckermr --- blog/2021-09-24-travis-security.md | 54 ++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 blog/2021-09-24-travis-security.md diff --git a/blog/2021-09-24-travis-security.md b/blog/2021-09-24-travis-security.md new file mode 100644 index 0000000000..feb5907f25 --- /dev/null +++ b/blog/2021-09-24-travis-security.md @@ -0,0 +1,54 @@ +--- +authors: + - beckermr +tags: [security] +--- +# Travis CI Security Incident + +On September 9, 2021 one of our core devs discovered that artifacts +building on Travis CI were being uploaded to our conda channel from PRs +running on forked repositories. A quick investigation revealed that +Travis CI was passing encrypted secrets to PR builds on forks. Further +examination of our logs and artifacts indicated that this had been +happening since about September 3, 2021. This security bug was +subsequently confirmed by Travis CI. See this +[CVE](https://nvd.nist.gov/vuln/detail/CVE-2021-41077) for more details +on this incident. **As far as we know, there were no actual exploits +against conda-forge which used this vulnerability.** + + + +## Our Response + +We took the following steps to respond to this incident. + +1. We immediately turned off all builds on Travis CI by suspending the + Travis CI GitHub App. +2. We immediately disclosed the bug to Travis CI through our contacts + there. +3. Once Travis CI indicated to us that they were ready, we rotated all + feedstock tokens and later our anaconda.org token for our staging + channel. The anaconda.org token for the main `conda-forge` channel + was never disclosed in this incident. Further, only ~70 feedstocks + had their tokens exposed in this incident. +4. We examined our artifacts and marked as broken any artifacts that + were uploaded from PRs. We think we found everything, but we are not + completely sure. Our criterion for marking things broken was more + generous than it needed to be. +5. We issued PRs to rebuild any broken artifacts via our bots. +6. We put in changes to `conda-smithy` to help prevent inadvertent + uploads of artifacts from PRs in the future. + +## Closing Thoughts & What can you do? + +I (MRB) want to recognize the quick work of our core dev team in +handling this incident. It goes without saying that the public nature of +`conda-forge`'s infrastructure carries risks. On the other hand, by +being public, anyone can look and verify our artifact builds. Security +for `conda-forge` is about reducing risk and we will continue to do our +best. + +Our best defense against security incidents in `conda-forge` is *you*! +Our feedstock maintainers are in the best position to notice incidents +and issues. Please responsibly report anything you find to us at +`condaforge+security@gmail.com`. From 45d20ad083ad2903b6074b55a86751dd0461f024 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:32:21 +0100 Subject: [PATCH 17/32] add blog/2021-11-03-tensorflow-gpu.md Co-authored-by: wolfv --- blog/2021-11-03-tensorflow-gpu.md | 114 ++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 blog/2021-11-03-tensorflow-gpu.md diff --git a/blog/2021-11-03-tensorflow-gpu.md b/blog/2021-11-03-tensorflow-gpu.md new file mode 100644 index 0000000000..e8d16c30ed --- /dev/null +++ b/blog/2021-11-03-tensorflow-gpu.md @@ -0,0 +1,114 @@ +--- +authors: + - wolfv +tags: [conda-forge] +image: https://i.imgur.com/pRdJaYw.png +--- + +# GPU enabled TensorFlow builds on conda-forge + +![Tensorflow on Anvil](https://i.imgur.com/pRdJaYw.png) + +Recently we've been able to add GPU-enabled TensorFlow builds to +conda-forge! This was quite a journey, with multiple contributors trying +different ways to convince the Bazel-based build system of TensorFlow to +build CUDA-enabled packages. But we managed, [and the pull request got +merged](https://github.com/conda-forge/tensorflow-feedstock/pull/157). + + + +We now have a configuration in place that creates CUDA-enabled +TensorFlow builds for all conda-forge supported configurations (CUDA +10.2, 11.0, 11.1, and 11.2+). Building out the CUDA packages requires +beefy machines -- on a 32 core machine it still takes around 3 hours to +build a single package. Our build matrix now includes 12 CUDA-enabled +packages & 3 CPU packages (because we need separate packages per Python +version). As one can imagine, this isn't easily possible on an average +"home computer". + +For this purpose, we have written an Ansible playbook that lets us boot +up cloud machines which then build the feedstock (using the +`build-locally.py` script). Thanks to the generous support +of OVH we were able to boot multiple 32-core virtual machines +simultaneously to build the different TensorFlow variants. + +We have open-sourced the [Ansible playbook in +GitHub](https://github.com/mamba-org/build-locally-ansible) and we're +working towards making it (more) generally useful for other long-running +builds! + +![Running 3 builds in parallel on 32 cores ... still takes around 3 hours to finish](https://i.imgur.com/nvV6izV.jpg){width="600px"} + +With the TensorFlow builds in place, conda-forge now has CUDA-enabled +builds for PyTorch and Tensorflow, the two most popular deep learning +libraries. + +We are still missing Windows builds for TensorFlow (CPU & CUDA, +unfortunately) and would love the community to help us out with that. +There is an open PR, but it probably needs some poking in Bazel to get +it to pass: +. + +We hope that these new GPU builds will enable many more packages to be +added to the conda-forge channel! We are already looking forward to the +2.6.2 and 2.7 releases of TensorFlow and to adding Windows support in +the future. We hope you enjoy this work. + +## Installation + +You can now select between GPU enabled (default) and CPU packages using +the `tensorflow-gpu` and `tensorflow-cpu` packages. Just run + +```bash +mamba install tensorflow-gpu -c conda-forge +# OR +conda install tensorflow-gpu -c conda-forge +``` + +When installing the `tensorflow` package, the package resolution will +now default to the GPU-enabled builds of tensorflow if the local machine +has a GPU (these builds can be identified by "cuda" at the beginning +of the version number). Note that GPU-enabled packages can also work on +CPU-only machines, but one would need to override the enviornment +variable `CONDA_OVERRIDE_CUDA` like below. This could be handy if you +are in a situation where your current node (e.g. login node) on an HPC +does not have GPUs, but the compute nodes with GPUs do not have internet +access. + +```bash +CONDA_OVERRIDE_CUDA="11.2" conda install tensorflow cudatoolkit>=11.2 -c conda-forge +# OR +CONDA_OVERRIDE_CUDA="11.2" mamba install tensorflow cudatoolkit>=11.2 -c conda-forge +``` + +Note that you should select the cudatoolkit version most appropraite for +your GPU; currently, we have "10.2", "11.0", "11.1", and "11.2" +builds available where the the "11.2" builds are compatible with all +cudatoolkits>=11.2. You could also force a specific version of +`cudatoolkit` by specifying it like above. Moreover, you could ensure +you get a sepcific build of tensorflow by appending the package name +like `tensorflow==2.7.0=cuda*` or `tensorflow==2.7.0=cuda112*`. If you +want the slimmer "cpu-only" package, then you can install +`tensorflow-cpu` directly or equivalently `tensorflow==2.7.0=cpu*`. At +the time of writing (February 2022), on a machine without a GPU, one +would always get the `-cpu` variant unless overriden like +above. This decision has been made to allow greater accessibility for +users with limited bandwidth and resources. + +## Thanks to + +- Mark Harfouche (@hmaarrfk) & Ista Zahn (@izahn) for their initial + work on the TensorFlow GPU builds, and all other TensorFlow + maintainers. Uwe Korn (@xhochy) for his work on the Bazel scripts & + TensorFlow -- and all the other maintainers of the [TensorFlow + feedstock](https://github.com/conda-forge/tensorflow-feedstock)! +- NVIDIA for pushing cudatoolkit and cudnn on conda-forge that makes + this possible +- OVH for their generous sponsoring of large build machines that we + could use to build the recipes +- Bloomberg for their sponsorship of QuantStack's involvement with + conda-forge +- Andreas Trawoger (@atrawog) for the Ansible scripts that this is + based on +- Thorsten Beier (@derthorsten) and Adrien Delsalle (@adriendelsalle) + for their contributions to the recipe From 1149d73a2b0e46dbbee3c5aab5796855ae5bb1a4 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:32:56 +0100 Subject: [PATCH 18/32] add blog/2022-08-26-outreachy-wrap-up-blog-2022.md Co-authored-by: ssurbhi560 --- .../2022-08-26-outreachy-wrap-up-blog-2022.md | 173 ++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 blog/2022-08-26-outreachy-wrap-up-blog-2022.md diff --git a/blog/2022-08-26-outreachy-wrap-up-blog-2022.md b/blog/2022-08-26-outreachy-wrap-up-blog-2022.md new file mode 100644 index 0000000000..6872eeb626 --- /dev/null +++ b/blog/2022-08-26-outreachy-wrap-up-blog-2022.md @@ -0,0 +1,173 @@ +--- +authors: + - name: Surbhi Sharma + title: Outreachy intern + url: https://github.com/ssurbhi560 + image_url: https://github.com/ssurbhi560.png +tags: [Outreachy] +--- + +# Outreachy 2022 Wrap-up Blog + +This blog is about my work during my Outreachy internship with +[conda-forge](https://conda-forge.github.io). Before that a little about +me - I am [Surbhi](https://github.com/ssurbhi560), an Outreachy intern +with conda-forge for the May-August 2022 cohort and I worked on +documenting the conda-forge ecosystem. + + + +The first issue I started working on when the internship began was +[Better anchoring of +announcements(#1611)](https://github.com/conda-forge/conda-forge.github.io/issues/1611). +The goal of this issue was to fix the anchor for each year and also +specific announcements in the Announcements section so as to provide +better navigation of the [Announcements +page](https://conda-forge.org/docs/user/announcements.html). This was +also the time when I was feeling quite overwhelmed and anxious since I +was just starting and was unsure if I would be able to give my best. But +thanks to my awesome mentors +[@Katherine](https://github.com/kathatherine) and +[@Matt](https://github.com/beckermr), who have always been so helpful, +I was able to have a good start. We solved this issue in two parts. The +first part was to add anchors to each year which is solved with [Improve +anchors for each year in the Announcements section. +(#1766)](https://github.com/conda-forge/conda-forge.github.io/pull/1766), +and the second part was adding anchors to each announcement and fixing +the RSS feed. + +The part of the documentation I focused on after completing the first +issue was [Maintainers' +Documentation](https://conda-forge.org/docs/maintainer/00_intro.html). +Many open issues needed to be taken care of to make the Maintainers' +documentation more useful and accessible for new maintainers. The open +tickets that we have worked on are: + +1. [Document extras feedstock-name + (#1769)](https://github.com/conda-forge/conda-forge.github.io/issues/1769) + and [Explain how to become a maintainer + (#1331)](https://github.com/conda-forge/conda-forge.github.io/issues/1331). + Closed with [add extra section-recipe maintainer and feedstock-name + (#1772)](https://github.com/conda-forge/conda-forge.github.io/pull/1772). + + > As we started with improving the Maintainer documentation, these + > were the issues we picked first to work on. The first issue was + > documenting how maintainers can use the "feedstock-name" directive + > for naming feedstocks differently than their package names in + > staged recipes. The second issue was documenting how one should + > become a package maintainer. + +2. [Add more steps in Improve the documentation section + (#1651)](https://github.com/conda-forge/conda-forge.github.io/issues/1651). + Closed with [Update "Improve the documentation" section with more + steps + (#1776)](https://github.com/conda-forge/conda-forge.github.io/pull/1776). + + > In this issue we added some additional steps for people who would + > like to start contributing to conda-forge, especially to + > documentation. + +3. [Add more information about Grayskull in the documentation itself + (#1655)](https://github.com/conda-forge/conda-forge.github.io/issues/1655). + Closed with + [#1777](https://github.com/conda-forge/conda-forge.github.io/pull/1777). + + > The documentation on Grayskull in docs lacked the answers to + > questions like what exactly Grayskull is and how one should use + > Grayskull to generate a recipe. With this issue, we added more + > documentation on Grayskull for users. + +4. [Clarify feedstock LICENSE.txt + (#803)](https://github.com/conda-forge/conda-forge.github.io/issues/803). + Closed with [Add Feedstock repository structure section + (#1786)](https://github.com/conda-forge/conda-forge.github.io/pull/1786). + + > The docs for contributing and maintaining conda recipes discuss + > when and how to distribute the license for a particular package. + > The auto-generated feedstock repositories also include a license + > in the root, which is different from the related package license. + > With this issue, we added documentation on the differences between + > those two licenses and briefly explained the feedstock repository + > structure. + +5. [DOC: New maintainer + (#1117)](https://github.com/conda-forge/conda-forge.github.io/issues/1117). + Closed with [Add "How regro-cf-autotick-bot create version update + PR?" section. + (#1788)](https://github.com/conda-forge/conda-forge.github.io/pull/1788). + + > With this issue we improved docs for the new maintainers and the + > working of the bot. A "How does `regro-cf-autotick-bot` create + > automatic version updates?" section was added to the docs, which + > explains the whole process of creating an automated version update + > PRs by bot. + +6. [Add Perl package hints to documentation + (#1536)](https://github.com/conda-forge/conda-forge.github.io/issues/1536). + Working on this + [#1790](https://github.com/conda-forge/conda-forge.github.io/pull/1790). + + > With this issue we added ​​packaging instructions for Perl packages + > with different build systems in the documentation. + +7. [DOC: Update documentation about tokens + (#1532)](https://github.com/conda-forge/conda-forge.github.io/issues/1532). + Closed with + [#1793](https://github.com/conda-forge/conda-forge.github.io/pull/1793). + + > Feedstocks have stopped storing encrypted tokens to upload + > packages, but outdated information on tokens was still present in + > the documentation. With this issue we removed the outdated + > information and also added a new section "How to update your + > feedstock token?" for maintainers. + +8. [Improve the documentation on arch_rebuild.txt + (#1668)](https://github.com/conda-forge/conda-forge.github.io/issues/1668). + Closed with + [#1794](https://github.com/conda-forge/conda-forge.github.io/pull/1794). + + > With this issue we improved the documentation on + > `arch_rebuild.txt` and how maintainers can add a feedstock to + > `arch-rebuild.txt` if it requires rebuilding with different + > architectures/platforms (such as ppc64le or aarch64). + +9. [Document migrators + (#1355)](https://github.com/conda-forge/conda-forge.github.io/issues/1355), + [Update migration docs + (#862)](https://github.com/conda-forge/conda-forge.github.io/issues/862), + and [document migrators + (#737)](https://github.com/conda-forge/conda-forge.github.io/issues/737) + . Closed with [Documenting Migrators and Migrations. + (#1801)](https://github.com/conda-forge/conda-forge.github.io/pull/1801). + + > With these, we added more documentation on migrations and + > migrators, which would help maintainers find answers to questions + > like - What is a migrator/migration, and what does it do? When can + > (and why would) they should reject a migration PR? And so on. + +10. [Add a section in docs on security aspects of conda-forge + (#1808)](https://github.com/conda-forge/conda-forge.github.io/issues/1808). + Closed with + [#1812](https://github.com/conda-forge/conda-forge.github.io/pull/1812). + + > Currently, information regarding the security considerations of + > conda-forge builds is scattered throughout the documentation, and + > therefore it is hard to find and read. With this issue, we will + > put all the information together in one place, which will help + > maintainers and users to know more about how conda-forge secures + > its packages and infrastructure. + +I met some wonderful people during the internship who helped me with all +my questions and doubts that I had. The experience during the internship +also helped me get better opportunities after completing the Outreachy +internship. I have learned so many things during the internship that it +would make a long list if I were to write all of those. But the most +important things I learned are: + +- The importance of documentation and how to write good documentation. +- The best practices to follow while writing documentation. +- More about conda-forge and packaging tools. + +And above all, Outreachy helped me feel more confident about my skills +and overcome the imposter syndrome I had before. Thanks again to my +awesome mentors and the kind people of the conda-forge community! :) From 96efe6809eff173c019afddbd4bb75a24529e602 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:33:24 +0100 Subject: [PATCH 19/32] add blog/2023-03-12-circle-ci-security-breach.md Co-authored-by: beckermr --- blog/2023-03-12-circle-ci-security-breach.md | 111 +++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 blog/2023-03-12-circle-ci-security-breach.md diff --git a/blog/2023-03-12-circle-ci-security-breach.md b/blog/2023-03-12-circle-ci-security-breach.md new file mode 100644 index 0000000000..b7aa8be183 --- /dev/null +++ b/blog/2023-03-12-circle-ci-security-breach.md @@ -0,0 +1,111 @@ +--- +authors: + - core +tags: [security] +--- +# CircleCI Security Incident + +In early January 2023, CircleCI informed us that they had a large +[security breach](https://circleci.com/blog/jan-4-2023-incident-report/) +where a third party had gained access to all the environment secrets +stored in the service. For `conda-forge`, these secrets are the API +token used to upload built packages to our staging area on +`anaconda.org` and the unique token we generate for each feedstock. The +feedstock tokens are used as part of our artifact staging process to +ensure that only the maintainers of a given feedstock can upload +packages built by that feedstock. Later in January, we were informed by +CircleCI that their security breach started on December 19, 2022, with +the bulk of the secrets being exfiltrated in plain text from their +servers a few days later. A malicious third-party with access to these +secrets could potentially upload compromised versions of any package on +`conda-forge` in a so-called "supply chain" attack. + + + +**We have produced a** [list of all possibly compromised +artifacts](https://raw.githubusercontent.com/conda-forge/conda-forge.github.io/main/misc/circle_ci_pkgs_dec2022_breach.json). + +**If you use** `conda-forge` **in very sensitive environments (which we +do not recommend!), please remove these artifacts from your system.** + +**To date, we know of no compromised artifacts in** `conda-forge`. + +**API tokens for the main** `conda-forge` **channel were never exposed +and remain secure to our knowledge.** + +## Our Response + +We took the following steps to respond to this incident. + +- We immediately started a token rotation of all of our feedstock + tokens and our staging area upload tokens as precautionary measures. + This token rotation hit a few bugs, but was completed as of January + 13, 2023. +- We produced a census of all packages uploaded between December 19, + 2022 and January 13, 2023. This data is available for download as a + [JSON + file](https://raw.githubusercontent.com/conda-forge/conda-forge.github.io/main/misc/circle_ci_pkgs_dec2022_breach.json). +- We examined all the artifacts built during this time period for the + [malicious + files](https://circleci.com/blog/jan-4-2023-incident-report/) listed + by CicleCI. We did not find any of those files in our artifacts. +- As detailed below, we have begun retooling our system for feedstock + tokens to be more robust and enable greater flexibility in our + response to incidents like this. +- We have begun systematically invalidating old tokens, + decommissioning old bots, and minimizing permissions of our current + tokens in order to further enhance `conda-forge`'s security. + +Rotating all of our tokens was taken as a precautionary measure. +Unfortunately, during this token rotation, one of our bots encountered a +bug which resulted in us losing the tokens for a very large fraction of +feedstocks. This situation resulted in an extended outage that lasted +about five days and was resolved on January 13, 2023, when the full +token rotation was completed. + +## What did we learn? + +We learned a few things about our system for feedstock tokens and +general maintenance of our CI service integrations. We probably should +have known them already, but here we are. + +- We used the same feedstock token across multiple CI services. This + limited our ability to immediately invalidate tokens associated with + a single CI service and exposed all services if any single service + had an incident. +- Our token system only allowed one valid token per feedstock. This + limitation means that we cannot recover from partially failed token + resets/rotations and are subject to race conditions during the + reset/rotation process that can cause failed package uploads. +- We need to be more proactive about cleaning up deprecated/removed CI + services. The use of CircleCI in `conda-forge` has been deprecated + for quite a while. Had we taken the time, and had the foresight, to + remove all of our secrets from CircleCI when it was deprecated, we + could have avoided the security incident all together. + +We have begun retooling our system for feedstock tokens in order to fix +the issues identified above and allow us to have more flexibility in +responding to security incidents. We have also started the process of +decommissioning several of our old CI services. These changes will take +time to implement. You can follow the progress on our various public +issue trackers. + +## Closing Thoughts & What can you do? + +We, the `conda-forge` core dev team, want to thank everyone for their +patience and support as we have responded to the various security +incidents and bugs detailed above. It goes without saying that the +public nature of `conda-forge`'s infrastructure carries risks. On the +other hand, by being public, anyone can look and verify our artifact +builds. Security for `conda-forge` is about reducing risk, and we will +continue to do our best. + +As a reminder, we do not recommend that you use `conda-forge` in +environments with sensitive information. `conda-forge`'s software is +built by our users and the core dev team cannot verify or guarantee that +this software is not malicious or has not been tampered with. + +Our best defense against security incidents in `conda-forge` is you! Our +feedstock maintainers are in the best position to notice incidents and +issues. Please responsibly report anything you find to us at +`condaforge+security@gmail.com`. From 14f460f50d4225ceeebcc8e1351907b7c7995a7e Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:34:08 +0100 Subject: [PATCH 20/32] add blog/2023-07-13-installer-security-fixes.md Co-authored-by: jezdez --- blog/2023-07-13-installer-security-fixes.md | 51 +++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 blog/2023-07-13-installer-security-fixes.md diff --git a/blog/2023-07-13-installer-security-fixes.md b/blog/2023-07-13-installer-security-fixes.md new file mode 100644 index 0000000000..dc60441e06 --- /dev/null +++ b/blog/2023-07-13-installer-security-fixes.md @@ -0,0 +1,51 @@ +--- +authors: + - core +tags: [security] +--- +# Security updates to our installers + +In June 2023, software engineers from +[Anaconda](https://www.anaconda.com) have reported a security issue in +the uninstallers that are included in the Windows versions of the +[miniforge and mambaforge +installers](https://github.com/conda-forge/miniforge), one of the main +ways to bootstrap conda-forge based conda and mamba distributions. + + + +The issue could, under specific conditions, unintentionally delete files +from your system during the uninstallation process. Anaconda has +published more details in the related +[blogpost](https://www.anaconda.com/blog/windows-installer-security-fix) +about the security fix for the miniconda and Anaconda Distribution +Windows installers as well. + +conda-forge is committed to fix the miniforge and mambaforge installers +equally to reduce the possible impact on conda-forge users and has +worked with Anaconda to mitigate the issue. + +- As such, we are strongly recommending all users of miniforge and + mambaforge to **update immediately** to the latest versions of + miniforge and mambaforge. Please download them from the [miniforge + repository's main page](https://github.com/conda-forge/miniforge) + or the [release specific + page](https://github.com/conda-forge/miniforge/releases/tag/23.1.0-4). +- For older versions, we are providing a **security patch for already + installed miniforge and mambaforge installations**. You can download + these from [release specific + page](https://github.com/conda-forge/miniforge/releases/tag/23.1.0-4) + as well, under the names + `Miniforge3-uninstaller-patch-Windows-x86_64.exe` and + `Mambaforge-uninstaller-patch-Windows-x86_64.exe`. + +:::note +To uninstall older versions of miniforge and mambaforge released before +July 1, 2023, please download the security patch fix prior to +uninstallation. +::: + +In order for this flaw to be triggered, a specific combination of +factors must align, including uninstallation permissions, system access, +usage of Windows, and an existing installation of miniforge or +mambaforge. From 4a0b3e65f47e23b77a5a3d5461683155d2a65126 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:35:20 +0100 Subject: [PATCH 21/32] add authors --- blog/authors.yml | 135 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 135 insertions(+) diff --git a/blog/authors.yml b/blog/authors.yml index d61b9a5c75..6f9b23658e 100644 --- a/blog/authors.yml +++ b/blog/authors.yml @@ -3,3 +3,138 @@ core: title: The conda-forge core team url: https://github.com/orgs/conda-forge/teams/core image_url: https://github.com/conda-forge.png +beckermr: + name: Matthew R. Becker + title: Member of conda-forge/core + url: https://github.com/beckermr + image_url: https://github.com/beckermr.png +chenghlee: + name: Cheng H. Lee + title: Member of conda-forge/core + url: https://github.com/chenghlee + image_url: https://github.com/chenghlee.png +chrisburr: + name: Chris Burr + title: Member of conda-forge/core + url: https://github.com/chrisburr + image_url: https://github.com/chrisburr.png +cj-wright: + name: Christopher J. 'CJ' Wright + title: Member of conda-forge/core + url: https://github.com/cj-wright + image_url: https://github.com/cj-wright.png +dopplershift: + name: Ryan May + title: Member of conda-forge/core + url: https://github.com/dopplershift + image_url: https://github.com/dopplershift.png +ericdill: + name: Eric Dill + title: Member of conda-forge/core + url: https://github.com/ericdill + image_url: https://github.com/ericdill.png +h-vetinari: + name: Axel Obermeier + title: Member of conda-forge/core + url: https://github.com/h-vetinari + image_url: https://github.com/h-vetinari.png +isuruf: + name: Isuru Fernando + title: Member of conda-forge/core + url: https://github.com/isuruf + image_url: https://github.com/isuruf.png +jakirkham: + name: John Kirkham + title: Member of conda-forge/core + url: https://github.com/jakirkham + image_url: https://github.com/jakirkham.png +jezdez: + name: Jannis Leidel + title: Member of conda-forge/core + url: https://github.com/jezdez + image_url: https://github.com/jezdez.png +loriab: + name: Lori A. Burns + title: Member of conda-forge/core + url: https://github.com/loriab + image_url: https://github.com/loriab.png +marcelotrevisani: + name: Marcelo Duarte Trevisani + title: Member of conda-forge/core + url: https://github.com/marcelotrevisani + image_url: https://github.com/marcelotrevisani.png +mariusvniekerk: + name: Marius van Niekerk + title: Member of conda-forge/core + url: https://github.com/mariusvniekerk + image_url: https://github.com/mariusvniekerk.png +mbargull: + name: Marcel Bargull + title: Member of conda-forge/core + url: https://github.com/mbargull + image_url: https://github.com/mbargull.png +minrk: + name: Min Ragan-Kelley + title: Member of conda-forge/core + url: https://github.com/minrk + image_url: https://github.com/minrk.png +mwcraig: + name: Matt Craig + title: Member of conda-forge/core + url: https://github.com/mwcraig + image_url: https://github.com/mwcraig.png +ocefpaf: + name: Filipe Pires Alvarenga Fernandes + title: Member of conda-forge/core + url: https://github.com/ocefpaf + image_url: https://github.com/ocefpaf.png +pkgw: + name: Peter K. G. Williams + title: Member of conda-forge/core + url: https://github.com/pkgw + image_url: https://github.com/pkgw.png +SylvainCorlay: + name: Sylvain Corlay + title: Member of conda-forge/core + url: https://github.com/SylvainCorlay + image_url: https://github.com/SylvainCorlay.png +synapticarbors: + name: Joshua Adelman + title: Member of conda-forge/core + url: https://github.com/synapticarbors + image_url: https://github.com/synapticarbors.png +sodre: + name: Patrick Sodré + title: Member of conda-forge/core + url: https://github.com/sodre + image_url: https://github.com/sodre.png +viniciusdc: + name: Vinicius Douglas Cerutti + title: Member of conda-forge/core + url: https://github.com/viniciusdc + image_url: https://github.com/viniciusdc.png +wolfv: + name: Wolf Vollprecht + title: Member of conda-forge/core + url: https://github.com/wolfv + image_url: https://github.com/wolfv.png +xhochy: + name: Uwe L. Korn + title: Member of conda-forge/core + url: https://github.com/xhochy + image_url: https://github.com/xhochy.png +kkraus14: + name: Keith Kraus + title: Member of conda-forge/core + url: https://github.com/kkraus14 + image_url: https://github.com/kkraus14.png +jaimergp: + name: Jaime Rodríguez-Guerra + title: Member of conda-forge/core + url: https://github.com/jaimergp + image_url: https://github.com/jaimergp.png +hmaarrfk: + name: Mark Harfouche + title: Member of conda-forge/core + url: https://github.com/hmaarrfk + image_url: https://github.com/hmaarrfk.png From 9426697d26e2efb3679f2281c28f9e36c292721a Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:36:22 +0100 Subject: [PATCH 22/32] remove demo blog post --- blog/2022-12-11-experiment.md | 18 ------------------ 1 file changed, 18 deletions(-) delete mode 100644 blog/2022-12-11-experiment.md diff --git a/blog/2022-12-11-experiment.md b/blog/2022-12-11-experiment.md deleted file mode 100644 index eeacff9768..0000000000 --- a/blog/2022-12-11-experiment.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -slug: docusaurus-experiment -title: Running an experiment with Docusaurus -authors: - - core -tags: [website] ---- - -This prototype website is built with Docusaurus. - - - -We are experimenting with this framework for two reasons: - -- Writing documentation about the conda-forge infrastructure _somewhere_ before we submit it to the official site. -- Seeing if Docusaurus could replace the current conda-forge website. - -This blog post is just a demonstration of how this would look like. From a19d1dccd624420462ffad629bb837ed597a3180 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 3 Nov 2023 21:29:31 +0100 Subject: [PATCH 23/32] pre-commit --- blog/2020-02-04-GSoC.md | 28 ++--- blog/2020-03-05-grayskull.md | 32 ++--- blog/2020-03-10-pypy.md | 28 ++--- blog/2020-07-06-scipy-bof.md | 22 ++-- blog/2020-10-29-macos-arm64.md | 2 +- blog/2020-12-26-year-in-review.md | 80 ++++++------ blog/2021-02-02-Outreachy.md | 114 +++++++++--------- blog/2021-06-16-graykull-step-by-step.md | 2 +- blog/2021-09-24-travis-security.md | 3 +- blog/2021-11-03-tensorflow-gpu.md | 30 ++--- .../2022-08-26-outreachy-wrap-up-blog-2022.md | 6 +- blog/2023-03-12-circle-ci-security-breach.md | 63 +++++----- blog/2023-07-13-installer-security-fixes.md | 27 +++-- 13 files changed, 220 insertions(+), 217 deletions(-) diff --git a/blog/2020-02-04-GSoC.md b/blog/2020-02-04-GSoC.md index a3a8ffec6e..0995eb1514 100644 --- a/blog/2020-02-04-GSoC.md +++ b/blog/2020-02-04-GSoC.md @@ -25,7 +25,7 @@ and user interactions on GitHub. Want to be a part of the team? Great! Take a look at the projects below and get in touch with us on GitHub! You can check the [GSoC -label]() for a +label](https://github.com/regro/cf-scripts/labels/GSOC) for a detailed listing of the issues that need work. 1. Maintenance and Refactoring @@ -34,24 +34,24 @@ detailed listing of the issues that need work. are great for people with a range of experience from beginners to true code ninjas. - - Integration Testing for the Autotick Bot + - Integration Testing for the Autotick Bot - > Run true integration testing on a copy of the graph to better - > test code changes and improve our CI process. - > - > **Issue**: - > - > **Experience Level**: advanced + > Run true integration testing on a copy of the graph to better + > test code changes and improve our CI process. + > + > **Issue**: + > + > **Experience Level**: advanced - - Work on the "code hardening" - [Milestone]() + - Work on the "code hardening" + [Milestone](https://github.com/regro/cf-scripts/milestone/4) - Address any of the issues in the milestone above related to code - refactoring and cleanup. + Address any of the issues in the milestone above related to code + refactoring and cleanup. - **Issues**: + **Issues**: - **Experience Level**: beginner to advanced + **Experience Level**: beginner to advanced 2. Automated PRs for Globally Pinned Packages diff --git a/blog/2020-03-05-grayskull.md b/blog/2020-03-05-grayskull.md index b77d9498de..ebff241087 100644 --- a/blog/2020-03-05-grayskull.md +++ b/blog/2020-03-05-grayskull.md @@ -261,16 +261,16 @@ extra: ### Major differences -| Attribute | Grayskull (0.2.1) | Skeleton (3.18.11) | -| --------- | ----------------- | ------------------ | - **Command** | ✅ grayskull pypi pytest | ✅ conda skeleton pypi pytest | - **Time** | ✅ 4 seconds | ❌ 31 seconds | - **License** | ✅ Added the license file and license type correctly | ❗️ Added just the license type | - **Host Requirements** | ✅ Added correctly all the host requirements | ❌ it didn't add the correctly the host dependencies. It added unnecessary dependencies and it is missing quite a few of them necessary to build the package | - **Run Requirements** | ✅ Missing just setuptools from the host requirements (but this dependency is not defined on pytest package) | ❌ incorrect dependencies added to the project | - **Selectors** | ✅ Skipping correctly Python 2 and added selectors for windows and python versions | ❌ it didn't add any information regarding selectors. Actually conda-build added wrong information which will result in a broken recipe. For example 'sys_platform == win32' was added which is a wrong format for conda recipes | - **Entry points** | ✅ Added all entry points correctly | ❌ No entry points | - **Does it build?** | ✅ YES | ❌ NO | +| Attribute | Grayskull (0.2.1) | Skeleton (3.18.11) | +| --------------------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Command** | ✅ grayskull pypi pytest | ✅ conda skeleton pypi pytest | +| **Time** | ✅ 4 seconds | ❌ 31 seconds | +| **License** | ✅ Added the license file and license type correctly | ❗️ Added just the license type | +| **Host Requirements** | ✅ Added correctly all the host requirements | ❌ it didn't add the correctly the host dependencies. It added unnecessary dependencies and it is missing quite a few of them necessary to build the package | +| **Run Requirements** | ✅ Missing just setuptools from the host requirements (but this dependency is not defined on pytest package) | ❌ incorrect dependencies added to the project | +| **Selectors** | ✅ Skipping correctly Python 2 and added selectors for windows and python versions | ❌ it didn't add any information regarding selectors. Actually conda-build added wrong information which will result in a broken recipe. For example 'sys_platform == win32' was added which is a wrong format for conda recipes | +| **Entry points** | ✅ Added all entry points correctly | ❌ No entry points | +| **Does it build?** | ✅ YES | ❌ NO | In the case of `noarch: python`, Grayskull is smart enough to detect when the recipe supports it, which is not done by Skeleton. It is @@ -348,11 +348,11 @@ grayskul pypi pytest requests=2.21.0 colorama ## Future plans -> - For the next major version (1.0.0) it is planned to add the -> functionality to be able to load the recipe and update just parts -> of it; -> - Generate Conda recipes using CRAN (R) channel (2.0.0); -> - Generate Conda recipes using Conan (C++) channel (3.0.0); +> - For the next major version (1.0.0) it is planned to add the +> functionality to be able to load the recipe and update just parts +> of it; +> - Generate Conda recipes using CRAN (R) channel (2.0.0); +> - Generate Conda recipes using Conan (C++) channel (3.0.0); ## Issues @@ -361,7 +361,7 @@ the repository: . Contributions are very welcome! :) ------------------------------------------------------------------------- +--- This work was possible thanks to the [NumFOCUS](https://numfocus.org/) Small Development Grant program. diff --git a/blog/2020-03-10-pypy.md b/blog/2020-03-10-pypy.md index dd2a2f9ea3..dea44468a0 100644 --- a/blog/2020-03-10-pypy.md +++ b/blog/2020-03-10-pypy.md @@ -11,10 +11,10 @@ environment Supported platforms are, -> - Linux-x86_64 (glibc 2.12 or newer) -> - OSX-x86_64 (OSX 10.9 or newer) -> - Linux-aarch64 (glibc 2.17 or newer) -> - Linux-ppc64le (glibc 2.17 or newer) +> - Linux-x86_64 (glibc 2.12 or newer) +> - OSX-x86_64 (OSX 10.9 or newer) +> - Linux-aarch64 (glibc 2.17 or newer) +> - Linux-ppc64le (glibc 2.17 or newer) @@ -74,9 +74,9 @@ a dependency on python_abi. For eg, ```yaml requirements: host: - - python 3.6 + - python 3.6 run: - - python + - python ``` is rendered as, @@ -84,10 +84,10 @@ is rendered as, ```yaml requirements: host: - - python 3.6.10 h9d8adfe_1009_cpython + - python 3.6.10 h9d8adfe_1009_cpython run: - - python >=3.6,<3.7.0a0 - - python_abi 3.6 *_cp36m + - python >=3.6,<3.7.0a0 + - python_abi 3.6 *_cp36m ``` whereas, @@ -95,9 +95,9 @@ whereas, ```yaml requirements: host: - - python 3.6 *_73_pypy + - python 3.6 *_73_pypy run: - - python + - python ``` is rendered as, @@ -105,10 +105,10 @@ is rendered as, ```yaml requirements: host: - - python 3.6.9 0_73_pypy + - python 3.6.9 0_73_pypy run: - - python >=3.6,<3.7.0a0 - - python_abi 3.6 *_pypy36_pp73 + - python >=3.6,<3.7.0a0 + - python_abi 3.6 *_pypy36_pp73 ``` Note that the PyPy ABI tag has `pp73` at the end which diff --git a/blog/2020-07-06-scipy-bof.md b/blog/2020-07-06-scipy-bof.md index f76d56cd95..bff8d5619b 100644 --- a/blog/2020-07-06-scipy-bof.md +++ b/blog/2020-07-06-scipy-bof.md @@ -20,23 +20,23 @@ seeking new viewpoints and opinions! ### State of Packaging -- What is the state of conda-forge today? -- What improvements are in the works? +- What is the state of conda-forge today? +- What improvements are in the works? ### New features/ideas -- What is something that's not in the works that you'd like to see - updated/improved? -- What are ways that folks who are interested but new to the community - can get involved? +- What is something that's not in the works that you'd like to see + updated/improved? +- What are ways that folks who are interested but new to the community + can get involved? ### Maintenance -- What is the maintenance burden and what pain points does it create? -- What are strategies to mitigate this burden? +- What is the maintenance burden and what pain points does it create? +- What are strategies to mitigate this burden? ### Questions to the audience -- What are your pain points when engaging with conda forge? -- What is something new you'd like to see happen? -- What feedback do audience members have? +- What are your pain points when engaging with conda forge? +- What is something new you'd like to see happen? +- What feedback do audience members have? diff --git a/blog/2020-10-29-macos-arm64.md b/blog/2020-10-29-macos-arm64.md index c943a954b9..a87014ffa6 100644 --- a/blog/2020-10-29-macos-arm64.md +++ b/blog/2020-10-29-macos-arm64.md @@ -131,7 +131,7 @@ conda smithy rerender ``` For python packages, add one or more of the following to -`recipe/meta.yaml` as needed, noting that you *must* only add +`recipe/meta.yaml` as needed, noting that you _must_ only add `numpy`, `cython`, and/or `pybind11` if they are used in `host:` as well, diff --git a/blog/2020-12-26-year-in-review.md b/blog/2020-12-26-year-in-review.md index 6224306ed9..8f9c279bb7 100644 --- a/blog/2020-12-26-year-in-review.md +++ b/blog/2020-12-26-year-in-review.md @@ -14,53 +14,53 @@ the big accomplishments our community has made this year. The `conda-forge` community has grown immensely this year. Here are some numbers to help give you an idea of the scale of our growth. -- The community has added 3,751 new, unique `conda` packages this - year, along with a corresponding number of new feedstocks. -- For the majority of 2020, the `conda-forge` channel on - `anaconda.org` exceeded 100 million downloads per month. -- In July of 2020, the `conda-forge` channel passed 2 billion total, - all-time downloads. -- We've grown our core developer community, adding seven new members - to the `conda-forge` Core team and at least two members to the - `staged-recipes` team. -- We now have over 2,500 recipe maintainers in the `conda-forge` - GitHub organization. +- The community has added 3,751 new, unique `conda` packages this + year, along with a corresponding number of new feedstocks. +- For the majority of 2020, the `conda-forge` channel on + `anaconda.org` exceeded 100 million downloads per month. +- In July of 2020, the `conda-forge` channel passed 2 billion total, + all-time downloads. +- We've grown our core developer community, adding seven new members + to the `conda-forge` Core team and at least two members to the + `staged-recipes` team. +- We now have over 2,500 recipe maintainers in the `conda-forge` + GitHub organization. ## Big New Features We've also shipped a ton of big updates to our core infrastructure this year. These updates include -- `PyPy` **support**: We added support for `PyPy` 3.6 and now supply - one of the biggest stacks of `PyPy`-enabled packages in the `PyPy` - ecosystem. -- **automerge**: We now support the automatic merging of PRs on - feedstocks using the `automerge` label or through an opt-in setting - in the `conda-forge.yml`. -- `R` **4.0 migration**: This migration was the first one to use our - `automerge` infrastructure at scale. With it, we completed a - complete rebuild/upgrade of the `R` ecosystem in about a week. -- `Python` **updates**: We deprecated `Python` 2.7, completed the - `Python` 3.8 migration, and got about 75% of the way through the - `Python` 3.9 migration. -- **compiler upgrades**: We upgraded our compiler infrastructure to - `GCC` 9 and `clang` 11. -- **CentOS 7 and CentOS 6 EOL**: We shipped an option to enable our - compilers to use the CentOS 7 `sysroot` in preparation for the - CentOS 6 EOL. We hope to complete the move to CentOS 7 early next - year. -- **miniforge**: We built our own standalone, `miniconda`-like - installers. These support a broad range of platforms, including - `osx-arm64` and `linux-aarch64`. -- **standalone Windows stack**: We fully decoupled our Windows recipes - from the `defaults` channel by rebuilding the `msys2` recipes. -- **Apple silicon support**: We added support for Apple silicon with - our `osx-arm64` platform. This platform is our first one to use a - fully cross-compiled infrastructure. -- **CUDA support**: We added support for building CUDA packages on - windows and added CUDA 11.0 support. +- `PyPy` **support**: We added support for `PyPy` 3.6 and now supply + one of the biggest stacks of `PyPy`-enabled packages in the `PyPy` + ecosystem. +- **automerge**: We now support the automatic merging of PRs on + feedstocks using the `automerge` label or through an opt-in setting + in the `conda-forge.yml`. +- `R` **4.0 migration**: This migration was the first one to use our + `automerge` infrastructure at scale. With it, we completed a + complete rebuild/upgrade of the `R` ecosystem in about a week. +- `Python` **updates**: We deprecated `Python` 2.7, completed the + `Python` 3.8 migration, and got about 75% of the way through the + `Python` 3.9 migration. +- **compiler upgrades**: We upgraded our compiler infrastructure to + `GCC` 9 and `clang` 11. +- **CentOS 7 and CentOS 6 EOL**: We shipped an option to enable our + compilers to use the CentOS 7 `sysroot` in preparation for the + CentOS 6 EOL. We hope to complete the move to CentOS 7 early next + year. +- **miniforge**: We built our own standalone, `miniconda`-like + installers. These support a broad range of platforms, including + `osx-arm64` and `linux-aarch64`. +- **standalone Windows stack**: We fully decoupled our Windows recipes + from the `defaults` channel by rebuilding the `msys2` recipes. +- **Apple silicon support**: We added support for Apple silicon with + our `osx-arm64` platform. This platform is our first one to use a + fully cross-compiled infrastructure. +- **CUDA support**: We added support for building CUDA packages on + windows and added CUDA 11.0 support. ------------------------------------------------------------------------- +--- We know that this year has been extremely difficult for so many of our community members and that the fantastic success of `conda-forge` would diff --git a/blog/2021-02-02-Outreachy.md b/blog/2021-02-02-Outreachy.md index 3a28d21f22..6405eb4cf8 100644 --- a/blog/2021-02-02-Outreachy.md +++ b/blog/2021-02-02-Outreachy.md @@ -28,7 +28,7 @@ Information page to learn more about eligibility for Outreachy. 3. Begin by contributing to the project by looking at our [issues page](https://github.com/conda-forge/conda-forge.github.io/issues). As you make contributions, record them on the Outreachy site. -4. Once you have made a few contributions, begin to write your application. +4. Once you have made a few contributions, begin to write your application. Ask the mentors to review the application before you submit it. ## Participant Expectations @@ -79,10 +79,10 @@ discussions. Please introduce yourself on the public project chat: - Gitter - [Follow this -link](https://gitter.im/conda-forge/conda-forge.github.io) to join this -project's public chat. + link](https://gitter.im/conda-forge/conda-forge.github.io) to join this + project's public chat. - Outreachy mentors will often be in the community public chat. The -project mentor's usernames are: `@viniciusdc`. + project mentor's usernames are: `@viniciusdc`. Here are some ready-to-go ways you can get started contributing on your own. @@ -95,15 +95,15 @@ own. As this project main goal is enhancing our current documentation, here are some preliminary tasks that you can inspect to get ideas: -- Write and improve the project's documentation; -- Link to duplicate issues, and suggest new issue labels, to keep - things organized; -- Go through open issues and suggest closing old ones; -- Ask clarifying questions on recently opened issues to move the - discussion forward; -- We also have issues regarding the main functionalities of our bot, - in particular the autotick bot. You could find some new information - or ideas for your contributing proposals. +- Write and improve the project's documentation; +- Link to duplicate issues, and suggest new issue labels, to keep + things organized; +- Go through open issues and suggest closing old ones; +- Ask clarifying questions on recently opened issues to move the + discussion forward; +- We also have issues regarding the main functionalities of our bot, + in particular the autotick bot. You could find some new information + or ideas for your contributing proposals. ## Good starter tasks: @@ -120,44 +120,44 @@ with the conda-forge environment. There are a few potential larger tasks that can come after a few smaller task contributions. These are included into our three main bases: -- [Users](https://conda-forge.org/docs/user/00_intro.html): In this - case, some good starter tasks are mainly checking the actual - contents of conda-forge users documentations, and ideas to better - express its contents. -- [Maintainers](https://conda-forge.org/docs/maintainer/00_intro.html): - There are a bunch of missed topics in this area, some information - have to be updated or rewritten for better understanding. Writing a - complete guide containing the actual steps and standard model for a - package recipe, building process (just a simple discussion) and how - conda-forge bot recognize defective licenses, recipes and packages - in general is highly welcomed. For further understanding of the - general system check [this - link](https://conda-forge.org/docs/maintainer/infrastructure.html). - - It can be funny to say, but lots of helpful ideas and bug - solutions appear on our gitter channel, so if you have time to - write guides about them... it's also an incredible task. -- [And organization](https://conda-forge.org/docs/orga/00_intro.html) - Our environment is changing everyday, because of that a lot of - information is lost in this process or even worse, not documented at - all! which leads to some difficulties inserting new members to - develop and further enhance the current process. - - The related work on this matter is highly welcomed and for a - better grasp of the situation you can start with this - [guideline](https://conda-forge.org/docs/orga/guidelines.html) - and read some of our posts in our - [blog](https://conda-forge.org/blog/blog/) - - Revitalizing ideas/projects for the conda-forge blog are - definitely welcomed; - - Currently we have some interesting projects going on inside our - ecosystem, which in return will need good documentation... Some - of the projects conda-forge is affiliated include the [auto-tick - bot](https://github.com/regro/cf-scripts), - [symbol-exporter](https://github.com/symbol-management/symbol-exporter) - and a new service we are eager to start developing is the - [distributed-bot](https://github.com/regro/cf-scripts/issues/1367). - All of them have a great coverage of subjects and lots of people - to help and give advice about the service structure and - functionalities. +- [Users](https://conda-forge.org/docs/user/00_intro.html): In this + case, some good starter tasks are mainly checking the actual + contents of conda-forge users documentations, and ideas to better + express its contents. +- [Maintainers](https://conda-forge.org/docs/maintainer/00_intro.html): + There are a bunch of missed topics in this area, some information + have to be updated or rewritten for better understanding. Writing a + complete guide containing the actual steps and standard model for a + package recipe, building process (just a simple discussion) and how + conda-forge bot recognize defective licenses, recipes and packages + in general is highly welcomed. For further understanding of the + general system check [this + link](https://conda-forge.org/docs/maintainer/infrastructure.html). + - It can be funny to say, but lots of helpful ideas and bug + solutions appear on our gitter channel, so if you have time to + write guides about them... it's also an incredible task. +- [And organization](https://conda-forge.org/docs/orga/00_intro.html) + Our environment is changing everyday, because of that a lot of + information is lost in this process or even worse, not documented at + all! which leads to some difficulties inserting new members to + develop and further enhance the current process. + - The related work on this matter is highly welcomed and for a + better grasp of the situation you can start with this + [guideline](https://conda-forge.org/docs/orga/guidelines.html) + and read some of our posts in our + [blog](https://conda-forge.org/blog/blog/) + - Revitalizing ideas/projects for the conda-forge blog are + definitely welcomed; + - Currently we have some interesting projects going on inside our + ecosystem, which in return will need good documentation... Some + of the projects conda-forge is affiliated include the [auto-tick + bot](https://github.com/regro/cf-scripts), + [symbol-exporter](https://github.com/symbol-management/symbol-exporter) + and a new service we are eager to start developing is the + [distributed-bot](https://github.com/regro/cf-scripts/issues/1367). + All of them have a great coverage of subjects and lots of people + to help and give advice about the service structure and + functionalities. ## Improving the documentation @@ -178,11 +178,11 @@ editing. The more manual process is as follows: -- Fork the conda-forge.github.io repository to your own GitHub user - account. -- Clone that fork onto your computer. -- Check out a new branch deriving from master to do your work. -- Make and commit your changes. -- Submit a pull request to the main repository proposing your changes. +- Fork the conda-forge.github.io repository to your own GitHub user + account. +- Clone that fork onto your computer. +- Check out a new branch deriving from master to do your work. +- Make and commit your changes. +- Submit a pull request to the main repository proposing your changes. Happy editing! diff --git a/blog/2021-06-16-graykull-step-by-step.md b/blog/2021-06-16-graykull-step-by-step.md index 2717d12eb0..f4ae9b9a13 100644 --- a/blog/2021-06-16-graykull-step-by-step.md +++ b/blog/2021-06-16-graykull-step-by-step.md @@ -25,7 +25,7 @@ to generate the recipe. Let us get started. 1. Install `grayskull` using `conda` through the `conda-forge` channel: - + ``` $ conda install -c conda-forge grayskull ``` diff --git a/blog/2021-09-24-travis-security.md b/blog/2021-09-24-travis-security.md index feb5907f25..0d7517789f 100644 --- a/blog/2021-09-24-travis-security.md +++ b/blog/2021-09-24-travis-security.md @@ -3,6 +3,7 @@ authors: - beckermr tags: [security] --- + # Travis CI Security Incident On September 9, 2021 one of our core devs discovered that artifacts @@ -48,7 +49,7 @@ being public, anyone can look and verify our artifact builds. Security for `conda-forge` is about reducing risk and we will continue to do our best. -Our best defense against security incidents in `conda-forge` is *you*! +Our best defense against security incidents in `conda-forge` is _you_! Our feedstock maintainers are in the best position to notice incidents and issues. Please responsibly report anything you find to us at `condaforge+security@gmail.com`. diff --git a/blog/2021-11-03-tensorflow-gpu.md b/blog/2021-11-03-tensorflow-gpu.md index e8d16c30ed..58ea6a3e64 100644 --- a/blog/2021-11-03-tensorflow-gpu.md +++ b/blog/2021-11-03-tensorflow-gpu.md @@ -97,18 +97,18 @@ users with limited bandwidth and resources. ## Thanks to -- Mark Harfouche (@hmaarrfk) & Ista Zahn (@izahn) for their initial - work on the TensorFlow GPU builds, and all other TensorFlow - maintainers. Uwe Korn (@xhochy) for his work on the Bazel scripts & - TensorFlow -- and all the other maintainers of the [TensorFlow - feedstock](https://github.com/conda-forge/tensorflow-feedstock)! -- NVIDIA for pushing cudatoolkit and cudnn on conda-forge that makes - this possible -- OVH for their generous sponsoring of large build machines that we - could use to build the recipes -- Bloomberg for their sponsorship of QuantStack's involvement with - conda-forge -- Andreas Trawoger (@atrawog) for the Ansible scripts that this is - based on -- Thorsten Beier (@derthorsten) and Adrien Delsalle (@adriendelsalle) - for their contributions to the recipe +- Mark Harfouche (@hmaarrfk) & Ista Zahn (@izahn) for their initial + work on the TensorFlow GPU builds, and all other TensorFlow + maintainers. Uwe Korn (@xhochy) for his work on the Bazel scripts & + TensorFlow -- and all the other maintainers of the [TensorFlow + feedstock](https://github.com/conda-forge/tensorflow-feedstock)! +- NVIDIA for pushing cudatoolkit and cudnn on conda-forge that makes + this possible +- OVH for their generous sponsoring of large build machines that we + could use to build the recipes +- Bloomberg for their sponsorship of QuantStack's involvement with + conda-forge +- Andreas Trawoger (@atrawog) for the Ansible scripts that this is + based on +- Thorsten Beier (@derthorsten) and Adrien Delsalle (@adriendelsalle) + for their contributions to the recipe diff --git a/blog/2022-08-26-outreachy-wrap-up-blog-2022.md b/blog/2022-08-26-outreachy-wrap-up-blog-2022.md index 6872eeb626..d5f2503809 100644 --- a/blog/2022-08-26-outreachy-wrap-up-blog-2022.md +++ b/blog/2022-08-26-outreachy-wrap-up-blog-2022.md @@ -164,9 +164,9 @@ internship. I have learned so many things during the internship that it would make a long list if I were to write all of those. But the most important things I learned are: -- The importance of documentation and how to write good documentation. -- The best practices to follow while writing documentation. -- More about conda-forge and packaging tools. +- The importance of documentation and how to write good documentation. +- The best practices to follow while writing documentation. +- More about conda-forge and packaging tools. And above all, Outreachy helped me feel more confident about my skills and overcome the imposter syndrome I had before. Thanks again to my diff --git a/blog/2023-03-12-circle-ci-security-breach.md b/blog/2023-03-12-circle-ci-security-breach.md index b7aa8be183..c99f15ffdb 100644 --- a/blog/2023-03-12-circle-ci-security-breach.md +++ b/blog/2023-03-12-circle-ci-security-breach.md @@ -3,6 +3,7 @@ authors: - core tags: [security] --- + # CircleCI Security Incident In early January 2023, CircleCI informed us that they had a large @@ -37,24 +38,24 @@ and remain secure to our knowledge.** We took the following steps to respond to this incident. -- We immediately started a token rotation of all of our feedstock - tokens and our staging area upload tokens as precautionary measures. - This token rotation hit a few bugs, but was completed as of January - 13, 2023. -- We produced a census of all packages uploaded between December 19, - 2022 and January 13, 2023. This data is available for download as a - [JSON - file](https://raw.githubusercontent.com/conda-forge/conda-forge.github.io/main/misc/circle_ci_pkgs_dec2022_breach.json). -- We examined all the artifacts built during this time period for the - [malicious - files](https://circleci.com/blog/jan-4-2023-incident-report/) listed - by CicleCI. We did not find any of those files in our artifacts. -- As detailed below, we have begun retooling our system for feedstock - tokens to be more robust and enable greater flexibility in our - response to incidents like this. -- We have begun systematically invalidating old tokens, - decommissioning old bots, and minimizing permissions of our current - tokens in order to further enhance `conda-forge`'s security. +- We immediately started a token rotation of all of our feedstock + tokens and our staging area upload tokens as precautionary measures. + This token rotation hit a few bugs, but was completed as of January + 13, 2023. +- We produced a census of all packages uploaded between December 19, + 2022 and January 13, 2023. This data is available for download as a + [JSON + file](https://raw.githubusercontent.com/conda-forge/conda-forge.github.io/main/misc/circle_ci_pkgs_dec2022_breach.json). +- We examined all the artifacts built during this time period for the + [malicious + files](https://circleci.com/blog/jan-4-2023-incident-report/) listed + by CicleCI. We did not find any of those files in our artifacts. +- As detailed below, we have begun retooling our system for feedstock + tokens to be more robust and enable greater flexibility in our + response to incidents like this. +- We have begun systematically invalidating old tokens, + decommissioning old bots, and minimizing permissions of our current + tokens in order to further enhance `conda-forge`'s security. Rotating all of our tokens was taken as a precautionary measure. Unfortunately, during this token rotation, one of our bots encountered a @@ -69,19 +70,19 @@ We learned a few things about our system for feedstock tokens and general maintenance of our CI service integrations. We probably should have known them already, but here we are. -- We used the same feedstock token across multiple CI services. This - limited our ability to immediately invalidate tokens associated with - a single CI service and exposed all services if any single service - had an incident. -- Our token system only allowed one valid token per feedstock. This - limitation means that we cannot recover from partially failed token - resets/rotations and are subject to race conditions during the - reset/rotation process that can cause failed package uploads. -- We need to be more proactive about cleaning up deprecated/removed CI - services. The use of CircleCI in `conda-forge` has been deprecated - for quite a while. Had we taken the time, and had the foresight, to - remove all of our secrets from CircleCI when it was deprecated, we - could have avoided the security incident all together. +- We used the same feedstock token across multiple CI services. This + limited our ability to immediately invalidate tokens associated with + a single CI service and exposed all services if any single service + had an incident. +- Our token system only allowed one valid token per feedstock. This + limitation means that we cannot recover from partially failed token + resets/rotations and are subject to race conditions during the + reset/rotation process that can cause failed package uploads. +- We need to be more proactive about cleaning up deprecated/removed CI + services. The use of CircleCI in `conda-forge` has been deprecated + for quite a while. Had we taken the time, and had the foresight, to + remove all of our secrets from CircleCI when it was deprecated, we + could have avoided the security incident all together. We have begun retooling our system for feedstock tokens in order to fix the issues identified above and allow us to have more flexibility in diff --git a/blog/2023-07-13-installer-security-fixes.md b/blog/2023-07-13-installer-security-fixes.md index dc60441e06..9a20587278 100644 --- a/blog/2023-07-13-installer-security-fixes.md +++ b/blog/2023-07-13-installer-security-fixes.md @@ -3,6 +3,7 @@ authors: - core tags: [security] --- + # Security updates to our installers In June 2023, software engineers from @@ -25,19 +26,19 @@ conda-forge is committed to fix the miniforge and mambaforge installers equally to reduce the possible impact on conda-forge users and has worked with Anaconda to mitigate the issue. -- As such, we are strongly recommending all users of miniforge and - mambaforge to **update immediately** to the latest versions of - miniforge and mambaforge. Please download them from the [miniforge - repository's main page](https://github.com/conda-forge/miniforge) - or the [release specific - page](https://github.com/conda-forge/miniforge/releases/tag/23.1.0-4). -- For older versions, we are providing a **security patch for already - installed miniforge and mambaforge installations**. You can download - these from [release specific - page](https://github.com/conda-forge/miniforge/releases/tag/23.1.0-4) - as well, under the names - `Miniforge3-uninstaller-patch-Windows-x86_64.exe` and - `Mambaforge-uninstaller-patch-Windows-x86_64.exe`. +- As such, we are strongly recommending all users of miniforge and + mambaforge to **update immediately** to the latest versions of + miniforge and mambaforge. Please download them from the [miniforge + repository's main page](https://github.com/conda-forge/miniforge) + or the [release specific + page](https://github.com/conda-forge/miniforge/releases/tag/23.1.0-4). +- For older versions, we are providing a **security patch for already + installed miniforge and mambaforge installations**. You can download + these from [release specific + page](https://github.com/conda-forge/miniforge/releases/tag/23.1.0-4) + as well, under the names + `Miniforge3-uninstaller-patch-Windows-x86_64.exe` and + `Mambaforge-uninstaller-patch-Windows-x86_64.exe`. :::note To uninstall older versions of miniforge and mambaforge released before From 5f4fd36dc37278f0fdc48b01e34d64cdbd535d1e Mon Sep 17 00:00:00 2001 From: jaimergp Date: Fri, 3 Nov 2023 21:47:29 +0100 Subject: [PATCH 24/32] rename to lowercase --- blog/{2020-02-04-GSoC.md => 2020-02-04-gsoc.md} | 0 blog/{2020-07-11-R-4.md => 2020-07-11-r-4.md} | 0 blog/{2021-02-02-Outreachy.md => 2021-02-02-outreachy.md} | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename blog/{2020-02-04-GSoC.md => 2020-02-04-gsoc.md} (100%) rename blog/{2020-07-11-R-4.md => 2020-07-11-r-4.md} (100%) rename blog/{2021-02-02-Outreachy.md => 2021-02-02-outreachy.md} (100%) diff --git a/blog/2020-02-04-GSoC.md b/blog/2020-02-04-gsoc.md similarity index 100% rename from blog/2020-02-04-GSoC.md rename to blog/2020-02-04-gsoc.md diff --git a/blog/2020-07-11-R-4.md b/blog/2020-07-11-r-4.md similarity index 100% rename from blog/2020-07-11-R-4.md rename to blog/2020-07-11-r-4.md diff --git a/blog/2021-02-02-Outreachy.md b/blog/2021-02-02-outreachy.md similarity index 100% rename from blog/2021-02-02-Outreachy.md rename to blog/2021-02-02-outreachy.md From 9ca74ec0478b77b8f3076cb14f5b8c2490ce237d Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:00:56 +0100 Subject: [PATCH 25/32] upgrade to docusaurus v3 --- blog/2020-02-04-gsoc.md | 12 ++++++------ blog/2020-03-05-grayskull.md | 6 +++--- blog/2020-11-20-anaconda-tos.md | 4 ++-- blog/2021-11-03-tensorflow-gpu.md | 3 +-- 4 files changed, 12 insertions(+), 13 deletions(-) diff --git a/blog/2020-02-04-gsoc.md b/blog/2020-02-04-gsoc.md index 0995eb1514..85b371eaf8 100644 --- a/blog/2020-02-04-gsoc.md +++ b/blog/2020-02-04-gsoc.md @@ -39,7 +39,7 @@ detailed listing of the issues that need work. > Run true integration testing on a copy of the graph to better > test code changes and improve our CI process. > - > **Issue**: + > **Issue**: [regro/cf-scripts261](https://github.com/regro/cf-scripts/issues/261) > > **Experience Level**: advanced @@ -49,7 +49,7 @@ detailed listing of the issues that need work. Address any of the issues in the milestone above related to code refactoring and cleanup. - **Issues**: + **Issues**: [regro/cf-scripts milestone #4](https://github.com/regro/cf-scripts/milestone/4) **Experience Level**: beginner to advanced @@ -63,7 +63,7 @@ detailed listing of the issues that need work. automated infrastructure to propose the migration of the pin itself. The project here is to add this functionality to our infrastructure. - **Issue**: + **Issue**: [regro/cf-scripts#665](https://github.com/regro/cf-scripts/issues/665) **Experience Level**: advanced @@ -76,7 +76,7 @@ detailed listing of the issues that need work. `conda` index. We currently do not take this delay into account when issuing PRs in a migration. - **Issue**: + **Issue**: [regro/cf-scripts#595](https://github.com/regro/cf-scripts/issues/595) **Experience Level**: beginner @@ -89,7 +89,7 @@ detailed listing of the issues that need work. have determined that a suitable fraction of the packages effected by a migration have been properly rebuilt. - **Issue**: + **Issue**: [regro/cf-scripts#595](https://github.com/regro/cf-scripts/issues/595) **Experience Level**: moderate @@ -107,6 +107,6 @@ detailed listing of the issues that need work. and also scaling that process to the entire set of `conda-forge` packages. - **Issue**: + **Issue**: [regro/cf-scripts#664](https://github.com/regro/cf-scripts/issues/664) **Experience Level**: moderate diff --git a/blog/2020-03-05-grayskull.md b/blog/2020-03-05-grayskull.md index ebff241087..02bd2f516c 100644 --- a/blog/2020-03-05-grayskull.md +++ b/blog/2020-03-05-grayskull.md @@ -27,7 +27,7 @@ looking for packages on [PyPI](https://pypi.org/), and it is available on [PyPI](https://pypi.org/project/grayskull/) and [conda-forge](https://github.com/conda-forge/grayskull-feedstock). The GitHub repository for this package is: -. +[marcelotrevisani/grayskull](https://github.com/marcelotrevisani/grayskull). Before Grayskull, we just had `conda-build skeleton` to generate recipes for Python packages on PyPI. In all other aspects, the difference of @@ -356,8 +356,8 @@ grayskul pypi pytest requests=2.21.0 colorama ## Issues -Any problem, question, suggestions please feel free to open an issue on -the repository: . +Any problem, question, suggestions please feel free to open [an issue on +the repository](https://github.com/marcelotrevisani/grayskull/issues). Contributions are very welcome! :) diff --git a/blog/2020-11-20-anaconda-tos.md b/blog/2020-11-20-anaconda-tos.md index d25e8c208a..56a0b1405d 100644 --- a/blog/2020-11-20-anaconda-tos.md +++ b/blog/2020-11-20-anaconda-tos.md @@ -38,8 +38,8 @@ employees also provide help with maintaining some of the most complex package recipes in conda-forge. Note that Anaconda Inc has also said that part of the revenue from the -TOS change will be donated to OSS projects. (See this blog post: -). +TOS change will be donated to OSS projects. (see this [blog +post](https://www.anaconda.com/blog/sustaining-our-stewardship-of-the-open-source-data-science-community)). You should be aware that `conda-forge` is a part of NumFOCUS, and so it stands to benefit from the change in TOS, as do many other OSS projects. diff --git a/blog/2021-11-03-tensorflow-gpu.md b/blog/2021-11-03-tensorflow-gpu.md index 58ea6a3e64..76bdcb1867 100644 --- a/blog/2021-11-03-tensorflow-gpu.md +++ b/blog/2021-11-03-tensorflow-gpu.md @@ -46,8 +46,7 @@ libraries. We are still missing Windows builds for TensorFlow (CPU & CUDA, unfortunately) and would love the community to help us out with that. There is an open PR, but it probably needs some poking in Bazel to get -it to pass: -. +it to pass: [conda-forge/tensorflow-feedstock#111](https://github.com/conda-forge/tensorflow-feedstock/pull/111). We hope that these new GPU builds will enable many more packages to be added to the conda-forge channel! We are already looking forward to the From 757ddfef5778d6f2988931da609474976a75fdc7 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:43:01 +0100 Subject: [PATCH 26/32] fix image --- blog/2021-11-03-tensorflow-gpu.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/2021-11-03-tensorflow-gpu.md b/blog/2021-11-03-tensorflow-gpu.md index 76bdcb1867..9f15e7c655 100644 --- a/blog/2021-11-03-tensorflow-gpu.md +++ b/blog/2021-11-03-tensorflow-gpu.md @@ -37,7 +37,7 @@ GitHub](https://github.com/mamba-org/build-locally-ansible) and we're working towards making it (more) generally useful for other long-running builds! -![Running 3 builds in parallel on 32 cores ... still takes around 3 hours to finish](https://i.imgur.com/nvV6izV.jpg){width="600px"} +![Running 3 builds in parallel on 32 cores ... still takes around 3 hours to finish](https://i.imgur.com/nvV6izV.jpg) With the TensorFlow builds in place, conda-forge now has CUDA-enabled builds for PyTorch and Tensorflow, the two most popular deep learning From 349febdc728bd307ab56d52dea0adf8f040d8e6d Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:43:39 +0100 Subject: [PATCH 27/32] pre-commit --- blog/2020-11-20-anaconda-tos.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/2020-11-20-anaconda-tos.md b/blog/2020-11-20-anaconda-tos.md index 56a0b1405d..079f539763 100644 --- a/blog/2020-11-20-anaconda-tos.md +++ b/blog/2020-11-20-anaconda-tos.md @@ -38,7 +38,7 @@ employees also provide help with maintaining some of the most complex package recipes in conda-forge. Note that Anaconda Inc has also said that part of the revenue from the -TOS change will be donated to OSS projects. (see this [blog +TOS change will be donated to OSS projects. (see this [blog post](https://www.anaconda.com/blog/sustaining-our-stewardship-of-the-open-source-data-science-community)). You should be aware that `conda-forge` is a part of NumFOCUS, and so it stands to benefit from the change in TOS, as do many other OSS projects. From eaa04c1389e259d276c5cf953a77bf881f5710a6 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Sat, 4 Nov 2023 13:47:02 +0100 Subject: [PATCH 28/32] truncate here --- blog/2020-07-11-r-4.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/blog/2020-07-11-r-4.md b/blog/2020-07-11-r-4.md index 4c6ffb75ac..00ef06fcfc 100644 --- a/blog/2020-07-11-r-4.md +++ b/blog/2020-07-11-r-4.md @@ -12,6 +12,8 @@ while, the recent migration of `r-java` and its dependents gives a good opportunity to write a retrospective on the technical issues with large-scale migrations in `conda-forge` and how we solved them. + + The R 4.0 migration rebuilt every package in `conda-forge` that had `r-base` as a requirement, including more than 2200 feedstocks. A migration of this size in `conda-forge` faces several hurdles. First, From a69214b7a1224eb55989e8c7985b244f97e96b69 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 8 Jan 2024 12:22:50 +0100 Subject: [PATCH 29/32] enable blog plugin; add some extra redirects --- SECURITY.md | 4 +- docusaurus.config.js | 246 ++++++++++++++++++++++--------------------- 2 files changed, 128 insertions(+), 122 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 7859561712..b6aabfcb6e 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -95,9 +95,9 @@ The role responsible for each step is noted at the beginning. mailing list. - **Developer**: Merge the security fix PR - **Developer**: Release the package and/or deploy the fix as appropriate - - **Developer & Coordinator**: Draft a [blog post](https://github.com/conda-forge/blog) and other + - **Developer & Coordinator**: Draft a [blog post](/blog) and other announcement texts. This can be done in parallel with the previous steps, but consider using a - [private advisory](https://github.com/conda-forge/blog/security/advisories) for the text. + [private advisory](https://github.com/conda-forge/conda-forge.github.io/security/advisories) for the text. - **Coordinator**: Publish the security advisory on the announcement date. If applicable, GitHub will post the CVE to the MITRE database. - **Coordinator**: Publish the blog post and other announcements (Element chat room, Twitter, diff --git a/docusaurus.config.js b/docusaurus.config.js index 0ddd9f2d5c..7f0a98f953 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -16,7 +16,7 @@ const config = { onBrokenLinks: "throw", onBrokenMarkdownLinks: "throw", favicon: "img/favicon.ico", - trailingSlash: undefined, + trailingSlash: true, staticDirectories: ['static', 'static-sphinx'], // GitHub pages deployment config. @@ -56,9 +56,9 @@ const config = { // breadcrumbs: true, // ...editUrl, // }, - // blog: { - // showReadingTime: true, - // }, + blog: { + showReadingTime: true, + }, theme: { customCss: require.resolve("./src/css/custom.css"), }, @@ -66,122 +66,128 @@ const config = { ], ], plugins: [ - // [ - // "content-docs", - // /** @type {import('@docusaurus/plugin-content-docs').Options} */ - // ({ - // id: "community", - // path: "community", - // routeBasePath: "/community", - // breadcrumbs: false, - // }), - // ], - // [ - // "content-blog", - // /** @type {import('@docusaurus/plugin-content-blog').Options} */ - // ({ - // id: "news", - // path: "news", - // routeBasePath: "/docs/user/announcements.html", - // showReadingTime: false, - // blogSidebarTitle: "Latest news", - // blogSidebarCount: 20, - // }), - // ], + // [ + // "content-docs", + // /** @type {import('@docusaurus/plugin-content-docs').Options} */ + // ({ + // id: "community", + // path: "community", + // routeBasePath: "/community", + // breadcrumbs: false, + // }), + // ], // [ - // "@docusaurus/plugin-client-redirects", - // { - // createRedirects(existingPath) { - // // Sphinx site tends to use 00_intro.html instead of index.html, which messes routing - // if (existingPath.endsWith('/00_intro') || existingPath.endsWith('/00_intro/') || existingPath.endsWith('/00_intro.html')) { - // // Redirect from /docs/team/X to /community/X and /docs/support/X to /community/X - // return [ - // existingPath.replace('/00_intro', '/'), - // existingPath.replace('/00_intro.html', '/index.html'), - // existingPath.replace('/00_intro.html', '/'), - // ]; - // } - // return undefined; // Return a falsy value: no redirect created - // }, - // redirects: [ - // { - // from: "/blog/posts/2019-12-06-cfep09/", - // to: "/blog/2019/12/06/cfep09/", - // }, - // { - // from: "/blog/posts/2020-02-04-GSoC/", - // to: "/blog/2020/02/04/gsoc/", - // }, - // { - // from: "/blog/posts/2020-03-05-grayskull/", - // to: "/blog/2020/03/05/grayskull/", - // }, - // { - // from: "/blog/posts/2020-03-10-pypy/", - // to: "/blog/2020/03/10/pypy/", - // }, - // { - // from: "/blog/posts/2020-07-02-op-risk/", - // to: "/blog/2020/07/02/op-risk/", - // }, - // { - // from: "/blog/posts/2020-07-06-scipy-bof/", - // to: "/blog/2020/07/06/scipy-bof/", - // }, - // { - // from: "/blog/posts/2020-07-11-R-4/", - // to: "/blog/2020/07/11/r-4/", - // }, - // { - // from: "/blog/posts/2020-10-02-versions/", - // to: "/blog/2020/10/02/versions/", - // }, - // { - // from: "/blog/posts/2020-10-29-macos-arm64/", - // to: "/blog/2020/10/29/macos-arm64/", - // }, - // { - // from: "/blog/posts/2020-11-20-anaconda-tos/", - // to: "/blog/2020/11/20/anaconda-tos/", - // }, - // { - // from: "/blog/posts/2020-12-26-year-in-review/", - // to: "/blog/2020/12/26/year-in-review/", - // }, - // { - // from: "/blog/posts/2021-02-02-outreachy/", - // to: "/blog/2021/02/02/outreachy/", - // }, - // { - // from: "/blog/posts/2021-06-16-graykull-step-by-step/", - // to: "/blog/2021/06/16/graykull-step-by-step/", - // }, - // { - // from: "/blog/posts/2021-09-24-travis-security/", - // to: "/blog/2021/09/24/travis-security/", - // }, - // { - // from: "/blog/posts/2021-11-03-tensorflow-gpu/", - // to: "/blog/2021/11/03/tensorflow-gpu/", - // }, - // { - // from: [ - // "/blog/posts/2022-08-19-outreachy-wrap-up-blog-2022/", - // "/blog/2022/08/19/outreachy-wrap-up-blog-2022/", - // ], - // to: "/blog/2022/08/26/outreachy-wrap-up-blog-2022/", - // }, - // { - // from: "/blog/posts/2023-03-12-circle-ci-security-breach/", - // to: "/blog/2023/03/12/circle-ci-security-breach/", - // }, - // { - // from: "/blog/posts/2023-07-13-installer-security-fixes/", - // to: "/blog/2023/07/13/installer-security-fixes/", - // }, - // ], - // }, + // "content-blog", + // /** @type {import('@docusaurus/plugin-content-blog').Options} */ + // ({ + // id: "news", + // path: "news", + // routeBasePath: "/docs/user/announcements.html", + // showReadingTime: false, + // blogSidebarTitle: "Latest news", + // blogSidebarCount: 20, + // }), // ], + [ + "@docusaurus/plugin-client-redirects", + { + createRedirects(existingPath) { + var redirects = []; + if (existingPath.startsWith('/blog/tags/')) { + redirects.push(`/blog/blog/tag/${existingPath.slice(11)}`) + } + if (existingPath == '/blog/archive/') { + redirects.push("/blog/2023/"); + redirects.push("/blog/2022/"); + redirects.push("/blog/2021/"); + redirects.push("/blog/2020/"); + redirects.push("/blog/2019/"); + } + return redirects; + }, + redirects: [ + { + from: "/blog/blog/", + to: "/blog/", + }, + { + from: "/blog/posts/2019-12-06-cfep09/", + to: "/blog/2019/12/06/cfep09/", + }, + { + from: "/blog/posts/2020-02-04-GSoC/", + to: "/blog/2020/02/04/gsoc/", + }, + { + from: "/blog/posts/2020-03-05-grayskull/", + to: "/blog/2020/03/05/grayskull/", + }, + { + from: "/blog/posts/2020-03-10-pypy/", + to: "/blog/2020/03/10/pypy/", + }, + { + from: "/blog/posts/2020-07-02-op-risk/", + to: "/blog/2020/07/02/op-risk/", + }, + { + from: "/blog/posts/2020-07-06-scipy-bof/", + to: "/blog/2020/07/06/scipy-bof/", + }, + { + from: "/blog/posts/2020-07-11-R-4/", + to: "/blog/2020/07/11/r-4/", + }, + { + from: "/blog/posts/2020-10-02-versions/", + to: "/blog/2020/10/02/versions/", + }, + { + from: "/blog/posts/2020-10-29-macos-arm64/", + to: "/blog/2020/10/29/macos-arm64/", + }, + { + from: "/blog/posts/2020-11-20-anaconda-tos/", + to: "/blog/2020/11/20/anaconda-tos/", + }, + { + from: "/blog/posts/2020-12-26-year-in-review/", + to: "/blog/2020/12/26/year-in-review/", + }, + { + from: "/blog/posts/2021-02-02-outreachy/", + to: "/blog/2021/02/02/outreachy/", + }, + { + from: "/blog/posts/2021-06-16-graykull-step-by-step/", + to: "/blog/2021/06/16/graykull-step-by-step/", + }, + { + from: "/blog/posts/2021-09-24-travis-security/", + to: "/blog/2021/09/24/travis-security/", + }, + { + from: "/blog/posts/2021-11-03-tensorflow-gpu/", + to: "/blog/2021/11/03/tensorflow-gpu/", + }, + { + from: [ + "/blog/posts/2022-08-19-outreachy-wrap-up-blog-2022/", + "/blog/2022/08/19/outreachy-wrap-up-blog-2022/", + ], + to: "/blog/2022/08/26/outreachy-wrap-up-blog-2022/", + }, + { + from: "/blog/posts/2023-03-12-circle-ci-security-breach/", + to: "/blog/2023/03/12/circle-ci-security-breach/", + }, + { + from: "/blog/posts/2023-07-13-installer-security-fixes/", + to: "/blog/2023/07/13/installer-security-fixes/", + }, + ], + }, + ], ], themeConfig: @@ -205,7 +211,7 @@ const config = { position: "left", }, { - href: "https://conda-forge.org/blog/", + to: "/blog/", label: "Blog", position: "left", }, @@ -301,7 +307,7 @@ const config = { }, { label: "Blog", - href: "https://conda-forge.org/blog/", + to: "/blog/", }, { label: "Status", From 46180f79063dcc19e0ad92ef3888fa887f883a5b Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 8 Jan 2024 12:39:05 +0100 Subject: [PATCH 30/32] show more blog posts in the sidebar --- docusaurus.config.js | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docusaurus.config.js b/docusaurus.config.js index 7f0a98f953..f21c72aee6 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -58,6 +58,8 @@ const config = { // }, blog: { showReadingTime: true, + blogSidebarCount: 10, + ...editUrl, }, theme: { customCss: require.resolve("./src/css/custom.css"), From 53315170ca2bb313de2096f5741be82991973dd1 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 8 Jan 2024 19:00:46 +0100 Subject: [PATCH 31/32] ensure case is respected --- docusaurus.config.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docusaurus.config.js b/docusaurus.config.js index f21c72aee6..e581620b20 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -157,7 +157,7 @@ const config = { to: "/blog/2020/12/26/year-in-review/", }, { - from: "/blog/posts/2021-02-02-outreachy/", + from: ["/blog/posts/2021-02-02-outreachy/", "/blog/2021/02/02/Outreachy/"], to: "/blog/2021/02/02/outreachy/", }, { From ad68b6681f9c3e83ae54016298dc17031cc1c79d Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 8 Jan 2024 19:05:24 +0100 Subject: [PATCH 32/32] remove #readme anchor --- sphinx/src/maintainer/infrastructure.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sphinx/src/maintainer/infrastructure.rst b/sphinx/src/maintainer/infrastructure.rst index 86dd678a72..13665a669a 100644 --- a/sphinx/src/maintainer/infrastructure.rst +++ b/sphinx/src/maintainer/infrastructure.rst @@ -231,7 +231,7 @@ Automerge The automerge service uses the GitHub action in this `repo `__. This action runs out of a Docker `container `__ on the ``prod`` tag. See the -repo `README.md `__ for more details. PRs are automatically merged if they satisfy either +repo `README.md `__ for more details. PRs are automatically merged if they satisfy either of the two following sets of conditions: 1. are from the ``regro-cf-autotick-bot``, have ``[bot-automerge]`` in the title, all statuses are passing, and the feedstock allows automerge @@ -245,7 +245,7 @@ Rerendering The rerendering service is triggered by the Heroku app. It uses the GitHub action in this `repo `__. This action runs out of a Docker `container `__ on the ``prod`` tag. See the -repo `README.md `__ for more details. +repo `README.md `__ for more details. Skipping CI builds