diff --git a/_images/codecov-configure.png b/_images/codecov-configure.png new file mode 100644 index 0000000..4b49b39 Binary files /dev/null and b/_images/codecov-configure.png differ diff --git a/_images/codecov-github.png b/_images/codecov-github.png new file mode 100644 index 0000000..dda08fe Binary files /dev/null and b/_images/codecov-github.png differ diff --git a/_images/codecov-pr.png b/_images/codecov-pr.png new file mode 100644 index 0000000..0f9c08b Binary files /dev/null and b/_images/codecov-pr.png differ diff --git a/_images/codecov-projects.png b/_images/codecov-projects.png new file mode 100644 index 0000000..a6eeee2 Binary files /dev/null and b/_images/codecov-projects.png differ diff --git a/_images/codecov-token.png b/_images/codecov-token.png new file mode 100644 index 0000000..f5920f5 Binary files /dev/null and b/_images/codecov-token.png differ diff --git a/_images/github-pages.png b/_images/github-pages.png new file mode 100644 index 0000000..c79128b Binary files /dev/null and b/_images/github-pages.png differ diff --git a/_images/precommit-PR.png b/_images/precommit-PR.png new file mode 100644 index 0000000..ecd1303 Binary files /dev/null and b/_images/precommit-PR.png differ diff --git a/_sources/conda-forge-guide.rst.txt b/_sources/conda-forge-guide.rst.txt new file mode 100644 index 0000000..6c79e81 --- /dev/null +++ b/_sources/conda-forge-guide.rst.txt @@ -0,0 +1,199 @@ +:tocdepth: -1 + +================================== +How to release conda-forge package +================================== + +.. _create-feedstock: + +I already have a conda-forge feedstock. I want to release a new package version. How do I do that? +-------------------------------------------------------------------------------------------------- + +Please skip to :ref:`here ` + +I am new to conda-forge. How do I create a conda package? +--------------------------------------------------------- + +Here, you will learn how to release a conda package distributed through the conda-forge channel in 10 to 15 minutes. This guide assumes you are familiar with a basic clone, fork, and pull request workflow on GitHub. + +Overview +^^^^^^^^ + +The process is divided into three steps: + + +1. :ref:`Prepare recipe: ` You will learn to prepare package information in a file called ``meta.yaml`` using our group's cookiecutting template. The file serves as a recipe for building your conda package. The recipe contains the package version, the source code, the dependencies, the license, etc. + +2. :ref:`Upload therecipe: ` Once you have the ``meta.yaml`` generated, you will create a pull request the staged-recipe repository in the conda-forge repository `here `_ + +3. :ref:`Recipe review: ` One of the community members of conda-forge will review your ``meta.yaml`` and provide feedback. Once the recipe is approved, you will have a package available for ``conda install`` automatically, and you will have your own designated feedstock repository that contains ``meta.yaml`` in ``https://github.com/conda-forge/-feedstock``. + +.. _conda-forge-recipe-prepare: + +1. Prepare conda package recipe in ``meta.yaml`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To generate a package, we first need to generate a "recipe" for the package. The recipe contains the type of programming language, the package version, the source code, the dependencies, and license, etc. This recipe is stored in a file called ``meta.yaml``. + +Hence, in Step 1, we will generate ``meta.yaml`` using the Billinge group's template. See https://github.com/conda-forge/diffpy.utils-feedstock/blob/main/recipe/meta.yaml as an example of a ``meta.yaml`` used in production. + +If you are interested in learning more about each component within ``meta.yaml``, read :ref:`Appendix 1 ` located at the end of this document. + +1. Install ``cookiecutter`` via ``pip install cookiecutter`` and run ``cookiecutter https://github.com/billingegroup/staged-recipes-cookiecutter`` + +2. Answer the following questions. Default values in parentheses are used if no value is provided. + + :github_org: The GitHub organization name. For example, ``diffpy``. + + :repo_name: The name of the repository. + + :module_name: The name of the module. + + :version: The version of the package. + + :Select: Choose PyPI. PyPI's ``sdist`` containing requirements files, src/tests, and ``pyproject.toml`` + + :short_description: A short description of the project + + :full_description: A full description of the project + + :license_file: The license file that is located in your project repository. i.g., ``LICENSE.rst``. + + :maintainers: You may have multiple maintainers ``sbillinge, bobleesj`` or just ``sbillinge`` + + :build_requirements: copy ``requirements/build.txt`` from the project repo. It should be empty for pure Python packages, otherwise compilers will be required. + + :host_requirements: Copy the following for the ``python >=3.11, setuptools, setuptools-git-versioning >=2.0, pip``. Copy ``requirements/host.txt`` + + :runtime_requirements: copy ``requirements/conda.txt``: + + :testing_requirements: copy ``requirements/test.txt`` + +Now, you have ``recipes//meta.yaml`` generated. + +.. important:: + - For a pure python package, have you removed the ``build`` section under the ``requirements``? See https://github.com/conda-forge/diffpy.utils-feedstock/blob/main/recipe/meta.yaml for example. + + - Have you double-checked the license file name in ``meta.yaml`` against the license files in the project repository. If you are unsure, please confirm with the repository owner (Prof. Billinge). + + +.. _conda-forge-recipe-upload: + +2. Upload ``meta.yaml`` to conda-forge for initial review +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Fork https://github.com/conda-forge/staged-recipes and clone your forked repository + +2. cd into ``staged-recipes`` + +3. Create ``recipes//meta.yaml`` Ex) ``recipes/diffpy.srreal/meta.yaml`` + +4. Copy and paste the content of ``meta.yaml`` from Step 1. + +5. Create a new branch: ``git checkout -b `` + +6. Add and commit the changes: ``git add . && git commit -m "Committing recipe for conda-forge release of "`` + +7. Push the changes: ``git push -u origin `` + +8. Visit https://github.com/conda-forge/staged-recipes and create a PR. + +9. Read through the pre-filled text in the PR message and follow the instructions. + +10. After the CI passes, create a new comment: ``@conda-forge/help-python Hello Team, ready for review!`` + +.. _conda-forge-recipe-review: + + +3. Wait for recipe review +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Wait for a ``conda-forge`` volunteer reviewer to review your submission. It may take up to one week. + +2. Once the PR is merged by the reviewer (1) your package is available on conda-forge, and (2) a new repository will be created under https://github.com/conda-forge/package-name-feedstock/. Example: https://github.com/conda-forge/diffpy.structure-feedstock. + + +.. _conda-forge-feedstock-release: + +Use the conda-forge feedstock to release a new version +------------------------------------------------------ + +We release a new package once we have the ``version`` and ``SHA256`` sections in ``meta.yaml`` in ``https://github.com/conda-forge/-feedstock`` located in the ``main`` branch. The conda-forge team asks to only modify ``meta.yaml``. + +First, we will attain th ``SHA256`` value from `pypi.org `_: + +#. Visit the project on PyPI at ``https://pypi.org/project/`` + +#. Click ``Download files`` under ``Navigation`` + +#. Click ``view hashes`` under ``Source Distribution`` + +#. Copy the ``SHA256`` value + +#. Create a PR to the feedstock repository. + +#. If you haven't, fork and clone the forked feedstock repository. + +#. Run ``git checkout main && git pull upstream main`` to sync with the main branch. + +#. Run ``git checkout -b `` to create a new branch. + +#. Open ``recipe/meta.yaml``, modify ``set version`` and ``sha256``. + +#. Run ``git add recipe/meta.yaml && git commit -m "release: ready for "``. + +#. Run ``git push --set-upstream origin ``. + +#. Create a PR to ``main``, complete the relevant checklists generated in the PR comment. + +#. Wait for the CI to pass and tag Project Owner for review. + +#. Once the PR is merged, in 20 to 30 minutes, verify the latest conda-forge package version from the README badge or by visiting ``https://anaconda.org/conda-forge/``. i.e.g, ``https://anaconda.org/conda-forge/diffpy.utils``. + + +.. _conda-forge-pre-release: + +Appendix 1. How do I do pre-release? +------------------------------------------------------------------------------ + +Generate ``meta.yaml`` by following ``Step 1`` and ``Step 2`` under ``conda-forge: release for the first time`` above. Here are two differences required for pre-release: + +1. Create ``recipe/conda_build_config.yaml`` containing:: + + channel_targets: + - conda-forge _rc + +See an example here: https://github.com/conda-forge/diffpy.pdffit2-feedstock/blob/rc/recipe/conda_build_config.yaml + +1. Make a PR into ``rc`` instead of ``main``. Re-render once the PR is created. + +To install the pre-release build:: + + conda install -c conda-forge/label/_rc -c conda-forge + +For more, read the documentation for pre-release: https://conda-forge.org/docs/maintainer/knowledge_base/#pre-release-builds + +Appendix 2. Add a new admin to the conda-forge feedstock +-------------------------------------------------------- + +Check whether you are an admin listed in the ``meta.yaml`` in the feedstock repository. Create an issue with the title/comment: ``@conda-forge-admin, please add user @username``. Please see an example issue `here `_. + + +.. _meta-yaml-info: + +Appendix 3. Background info on ``meta.yml`` +------------------------------------------- + +The ``meta.yaml`` file contains information about dependencies, the package version, the license, the documentation link, and the maintainer(s) of the package. In ``meta.yaml``, there are 3 important keywords under the ``requirements`` section: ``build``, ``host``, and ``run`` that are used to specify dependencies. + +- ``build`` dependencies used for compiling but are not needed on the host where the package will be used. Examples include compilers, CMake, Make, pkg-config, etc. + +- ``host`` dependencies are required during the building of the package. Examples include setuptools, pip, etc. + +- ``run`` dependencies are required during runtime. Examples include matplotlib-base, numpy, etc. + +To avoid any confusion, there is a separate YAML section called ``build`` above the ``requirements`` section. This section is for setting up the entire operating system. + +For more information, please refer to the official documentation: https://conda-forge.org/docs/maintainer/adding_pkgs/#build-host-and-run + +.. _conda-forge-add-admin: diff --git a/_sources/conda_forge_guide.rst.txt b/_sources/conda_forge_guide.rst.txt deleted file mode 100644 index 7a0ad3b..0000000 --- a/_sources/conda_forge_guide.rst.txt +++ /dev/null @@ -1,177 +0,0 @@ -:tocdepth: -1 - -================================== -How to release conda-forge package -================================== - -.. _create-feedstock: - -I am new to conda-forge. How do I create a conda package? ---------------------------------------------------------- - -Here, you will learn how to release a conda package distributed through the conda-forge channel in 10-15 minutes. This guide assumes you are familiar with a basic clone, fork, and pull request (PR) workflow on GitHub. - -Step 1. Prepare ``meta.yaml``. See Appendix 1 to learn more about ``meta.yaml`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To generate a package, we first need to generate a "recipe" for the package. The recipe contains the type of programming language, the package version, the source code, the dependencies, and license, etc. This recipe is stored in a file called ``meta.yaml``. - -Hence, in Step 1, we will generate ``meta.yaml`` using the Billinge group's template. See https://github.com/conda-forge/diffpy.utils-feedstock/blob/main/recipe/meta.yaml as an example of a ``meta.yaml`` used in production. - -If you are interested in learning more about each component within ``meta.yaml``, read :ref:`Appendix 1 ` located at the end of this document. - -1. Install ``cookiecutter`` via ``pip install cookiecutter`` and run ``cookiecutter https://github.com/billingegroup/staged-recipes-cookiecutter`` - -2. Answer the following questions. Default values in parentheses are used if no value is provided. - - 1. github_org (diffpy): - - 2. module_name (diffpy.my_project): diffpy.srreal - - 3. repo_name (diffpy.srreal): - - 4. version (1.0.0): 1.3.1 - - 5. Select source 1 - PyPi or 2 - GitHub: 1 - - 1. Choose 1. Conda package can be built using PyPI's ``sdist`` containing requirements files, src/tests, and ``pyproject.toml`` - - 6. project_short_description (Python package for doing science.): - - 7. project_full_description (This is a Python package for doing science.): - - 8. license_file (LICENSE.rst): - - 9. recipe_maintainers (sbillinge,): - - 10. build_requirements (): - - 1. copy ``requirements/build.txt`` from the project repo. - - 2. Empty for pure Python packages, otherwise compilers will be required. - - 11. host_requirements (python >=3.11, setuptools, setuptools-git-versioning >=2.0, pip,): - - 1. copy ``requirements/host.txt`` - - 12. runtime_requirements (python >=3.11, numpy,): - - 1. copy ``requirements/conda.txt`` - - 13. testing_requirements (pip, pytest,): - 1. copy ``requirements/test.txt`` - - -Now, you have ``recipes//meta.yaml`` is generated. - -- [ ] For a pure python package, have you removed the ``build`` section under the ``requirements``? See https://github.com/conda-forge/diffpy.utils-feedstock/blob/main/recipe/meta.yaml for example. - -- [ ] Have you double-checked the license file name in ``meta.yaml`` against the license files in the project repository. If you are unsure, please confirm with the repository owner (Prof. Billinge). - - -Step 2. Upload ``meta.yaml`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -1. Fork https://github.com/conda-forge/staged-recipes and clone your forked repository - -2. cd into ``staged-recipes`` - -3. Create ``recipes//meta.yaml`` Ex) ``recipes/diffpy.srreal/meta.yaml`` - -4. Copy and paste the content of ``meta.yaml`` from Step 1. - -5. Create a new branch: ``git checkout -b `` - -6. Add and commit the changes: ``git add . && git commit -m "Committing recipe for conda-forge release of "`` - -7. Push the changes: ``git push -u origin `` - -8. Visit https://github.com/conda-forge/staged-recipes and create a PR. - -9. Read through the pre-filled text in the PR message and follow the instructions. - -10. After the CI passes, create a new comment: ``@conda-forge/help-python Hello Team, ready for review!`` - -Step 3. Wait for review and merge -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -1. Wait for a ``conda-forge`` volunteer reviewer to review your submission. It may take up to one week. - -2. Once the PR is merged by the reviewer (1) your package is available on conda-forge, and (2) a new repository will be created under https://github.com/conda-forge/package-name-feedstock/. Example: https://github.com/conda-forge/diffpy.structure-feedstock. - - -I already have a conda package. How do I distribute a new package version? --------------------------------------------------------------------------- - -This step assumes there is a new version of Python package released to PyPI. We will use that PyPI source code to generate a new conda package. - -Obtain the ``SHA256`` value from `pypi.org `_: - -1. Visit the project on PyPI at ``https://pypi.org/project/`` - -2. Click ``Download files`` under ``Navigation`` - -3. Click ``view hashes`` under ``Source Distribution`` - -4. Copy the ``SHA256`` value - -Create a PR to your conda-forge feedstock: - -1. Fork the feedstock repository i.g. https://github.com/conda-forge/diffpy.utils-feedstock. - -2. Clone the forked repository. - -3. Run ``git checkout main && git pull upstream main`` to sync with the main branch. - -4. Run ``git checkout -b `` to create a new branch. - -5. Open ``recipe/meta.yaml``, modify 1) ``set version`` and 2) ``sha256``. - -6. Run ``git add recipe/meta.yaml && git commit -m "Release "``. - -7. Run ``git push --set-upstream origin ``. - -8. Create a PR to ``main``, complete the relevant checklists generated in the PR comment, and modify ``meta.yaml`` as needed. - -9. Wait for the CI to pass and tag Prof. Billinge for review. - -10. Once the PR is merged, verify the latest conda-forge package version from the README badge. - -.. _conda-pre-release: - -I am familiar with the regular conda release process. How do I do pre-release? ------------------------------------------------------------------------------- - -Generate ``meta.yaml`` by following ``Step 1`` and ``Step 2`` under ``conda-forge: release for the first time`` above. Here are two differences required for pre-release: - -1. Create ``recipe/conda_build_config.yaml`` containing:: - - channel_targets: - - conda-forge _rc - -See an example here: https://github.com/conda-forge/diffpy.pdffit2-feedstock/blob/rc/recipe/conda_build_config.yaml - -1. Make a PR into ``rc`` instead of ``main``. Re-render once the PR is created. - -To install the pre-release build:: - - conda install -c conda-forge/label/_rc -c conda-forge - -For more, read the documentation for pre-release: https://conda-forge.org/docs/maintainer/knowledge_base/#pre-release-builds - -.. _appendix1: - -Appendix 1. Background info on ``meta.yml`` -------------------------------------------- - -The ``meta.yaml`` file contains information about dependencies, the package version, the license, the documentation link, and the maintainer(s) of the package. In ``meta.yaml``, there are 3 important keywords under the ``requirements`` section: ``build``, ``host``, and ``run`` that are used to specify dependencies. - -- ``build`` dependencies used for compiling but are not needed on the host where the package will be used. Examples include compilers, CMake, Make, pkg-config, etc. - -- ``host`` dependencies are required during the building of the package. Examples include setuptools, pip, etc. - -- ``run`` dependencies are required during runtime. Examples include matplotlib-base, numpy, etc. - -To avoid any confusion, there is a separate YAML section called ``build`` above the ``requirements`` section. This section is for setting up the entire operating system. - -For more information, please refer to the official documentation: https://conda-forge.org/docs/maintainer/adding_pkgs/#build-host-and-run diff --git a/_sources/cookiecutter-guide.rst.txt b/_sources/cookiecutter-guide.rst.txt new file mode 100644 index 0000000..0f96b8a --- /dev/null +++ b/_sources/cookiecutter-guide.rst.txt @@ -0,0 +1,411 @@ +:tocdepth: -1 + +.. index:: cookiecutter-guide + +.. _cookiecutter-title: + +=============================== +How to cookiecut Python package +=============================== + +.. _cookiecutter-installation: + +Installation +------------ + +To get started, install ``cookiecutter``, ``black``, and ``pre-commit`` in a new conda environment. Follow the steps below: + +Create a new environment named ``cookiecutter_env``: :: + + conda create -n cookiecutter_env python=3.13 + +Activate the environment: :: + + conda activate cookiecutter_env + +Install packages: :: + + pip install cookiecutter black pre-commit + +You are now ready to cookiecut your Python package! + +Overview +-------- + +We have divided the cookiecutting process into four workflows: + +1. :ref:`Pre-commit workflow: ` you will use automatic formatting tools to standardize your package with PEP8 before migrating it to the Billinge group's project structure with ``cookiecutter``. Then, the ``pre-commit`` library installed is used ensure the code is in good shape. You can skip this step if you are starting a new project. + +2. :ref:`Cookiecutting workflow: ` After your code is formatted, you will use the ``cookiecutter`` library to generate a new project inside the package directory. The new project contains dynamically filled templates based on your inputs such as repository name, license, and contributors. Then, you will move files from the old to the new structure using Git. + +3. :ref:`API documentation build workflow: ` Once you have cookiecuttered the package, you will use our Python script to automatically generate API documentation for your package and render the documentation locally. + +4. :ref:`Final sign-off: ` After you've checked the licenses, README, and documentation, you will host your package documentation online. Once you are done with this page, we will guide you on how to release your package on a separate page :ref:`here `. + +Tips and how to receive support +------------------------------- + +We offer the following ways to help guide you through the cookiecutting process: + +1. You may cross-check with the Billinge group's up-to-date cookiecuttered package, ``diffpy.utils``: https://github.com/diffpy/diffpy.utils. + +2. If you have any questions, first read the :ref:`FAQ ` for how to customize your package and certain design decisions in the cookiecutter template. + +3. After you've cross-checked and searched through the FAQ, please feel free to ask questions by creating an issue on the Cookiecutter repository `here `_. + +.. _cookiecutter-workflow-pre-commit: + +1. Pre-commit workflow +---------------------- + +.. Important:: Skip this section and go to :ref:`2. Cookiecutter main workflow ` if you are starting a new project! + +#. Fork and clone the repository. + +#. ``cd`` into the top-level directory of that project. + +#. Type ``git pull upstream main`` to sync with the main branch. + +#. Double-check that no bug-fix etc. pull-requests are waiting to be merged. If you are a member, check with the project repository owner if you are unsure. + +#. Create a new branch called ``black``. + +#. Create ``pyproject.toml``. Copy and paste the following to ``pyproject.toml``. + + .. code-block:: bash + + [tool.black] + line-length = 115 + include = '\.pyi?$' + exclude = ''' + /( + \.git + | \.hg + | \.mypy_cache + | \.tox + | \.venv + | \.rst + | \.txt + | _build + | buck-out + | build + | dist + + # The following are specific to Black, you probably don't want those. + | blib2to3 + | tests/data + )/ + +#. Run ``black src`` in your Terminal. If your source code is in a different directory, replace ``src`` with the appropriate directory path. This will automatically format your code to PEP8 standards given the line-length provided under ``line-length`` above in ``pyproject.toml``. + +#. Add and commit the automatic changes by ``black``. The commit message can be ``git commit -m "style: apply black to src directory with black configured in pyproject.toml"``. + +#. Run ``black .`` Here, you are running black across the entire package directory. Then, test your package with unit tests locally. + +#. git add and commit the changes. The commit message can be ``git commit -m "style: apply black to all files in the project directory"``. + +#. Create a pull request into ``main``. The pull request title can be ``cookiecut: Apply black to project directory with no manual edits``. + +#. After the ``black`` branch has been merged to ``main``, type ``git checkout main && git pull upstream main`` and create a new branch called ``precommit`` by typing ``git checkout -b precommit``. + +#. Copy and paste two files of ``.flake8`` `here `_ and ``.pre-commit-config.yaml`` `here `_ to your project directory. Cross-check with https://github.com/diffpy/diffpy.utils. + +#. Run ``pre-commit run --all-files`` in your Terminal. This will attempt to lint your code such as docstrings, extra spaces, across all file types such as ``.yml``, ``.md``, ``.rst``, etc. However, most likely, you will have to manually fix some of the errors raised by ``flake8``. + +#. Before manually editing, let's first take a look at the changes made by running ``git status`` to get an overview of the files modified and then by running ``git diff `` to see the specific changes. If you do not want the new changes, you can run ``git restore `` to revert the changes. + + .. note:: + + Q1. Do you want to prevent certain automatic modifications on files? You can add the folder or extension to the ``exclude`` section in ``.pre-commit-config.yaml`` with an example shown `here `_. + + Q2. Do you want to ignore certain spelling recommendations by Codespell? Please refer to this section in the FAQ :ref:`here `. + +#. At this point, you may have flake8 errors but we want to address them in a separate pull request. Hence, git add and commit and push the automatic changes made by ``precommit`` and create a pull request to ``main``. The commit message can be ``style: apply pre-commit without manual modification`` and the pull request title can be ``cookiecut: Apply pre-commit to project directory with no manual edits``. + +#. After the ``precommit`` branch has been merged to ``main``, run ``git checkout main && git pull upstream main`` and create a new branch called ``flake8`` by typing ``git checkout -b flake8``. If you have many flake8 errors and types, feel free to create one branch for each specific type of error, like ``flake8-length``. + +Here are some tips to reduce cognitive overload: + + 1. Start with easier error types to fix, such as line lengths and "module imported but not used", etc. + + 2. Create multiple PRs, each containing a specific theme (e.g., "Fix docstring line-length flake8 errors" using the ``flake8-length`` branch, etc.) to reduce cognitive overload for the reviewer. + + 3. If you are unsure, suppress the flake8 error by adding ``# noqa: `` at the end of the line. For example, ``import numpy as np # noqa: E000`` but make sure you create an issue for this so that you can revisit them after cookiecutting. + +For each `flake8` branch, create a PR request to ``main``. Since you are fixing flake8 errors, the commit message can be ``style: fix flake8 errors`` and the pull request title can be ``cookiecut: Fix flake8 errors``. In each PR, feel free to communicate the remaining flake8 issues in each pull request to track progress. + +Congratulations! You have successfully completed the pre-commit workflow. You may proceed to the section to now transform your package structure! + +.. _cookiecutter-workflow-main: + +1. Cookiecutter main workflow +----------------------------- + +If you are migrating an existing project, + +.. Attention:: Ensure no files are overwritten or lost. + + - Do NOT delete/remove any files before confirming that it is absolutely unnecessary. Create an issue or contact the maintainer. + + - Do NOT delete project-specific content such as project descriptions in README, license information, authors, tutorials, examples. + + If you are unsure, please ask for help. + +If you are here starting a new project, the :ref:`1, Pre-commit workflow ` section is completed in the project template. Visit `installation <_cookiecutter-installation>`_ section and download the dependencies that we will be using. + +1. Type ``cookiecutter https://github.com/billingegroup/cookiecutter`` inside the package directory. + +2. Answer the questions as the following -- note that (default) means to hit enter without modifying anything: + +:github_org: The GitHub organization name or owner's GitHub username. e.g., ``Billingegroup``. + +:keywords: The keywords of the project. Each word is separated by a comma and a space. e.g., ``pdf, diffraction, neutron, x-ray``. The keywords may be found in ``pyproject.toml`` or ``setup.py``. + +:project_name: The name of the project e.g., ``my-package``. For a namespace package, use e.g., ``diffpy.my-package``. + +:package_dist_name: The name in the package distribution in PyPI and conda-forge .If your package name contains ``_``, replace it with ``-``. e.g., ``my-package``. For a namespace package, use e.g., ``diffpy.my-package``. + +:package_dir_name: The name of the package directory. It must be lowercase e.g., ``my_package`` so that it can be imported as ``import my_package`` in a Python script. + +:repo_name: The repository name of the project displayed on GitHub. It should be identical as the project_name. + +:min_python: The minimum Python version. The default is ``3.11``. + +:max_python: The maximum Python version. The default is ``3.13``. + +:have_c_code: Whether the package require C/C++ code that requires building the package. For pure Python packages, this is ``False``. The default is ``False``. + +:headless_GUI_test: Run headless testing in GitHub CI. If your package does not contain GUI, the default is ``False``. + +:workflow_version: Version of the reuseuable workflow to use. ``v0`` is the default. + +.. Important:: Skip the rest of Cookiecutter main workflow if you are starting a new project! Proceed to the :ref:`3. API documentation workflow` below. Otherwise, please continue! + +3. cd into the new ``diffpy./`` directory (e.g., in our example ``pwd`` would return ``~/dev/diffpy.pdfmorph/diffpy.pdfmorph``) (we will refer to the nested directory as the "**cookiecutter**" directory and ``~/dev/diffpy.pdfmorph/`` as the "**main**" directory). + +4. Type ``ls -als`` (if you have the alias, this is ``ll``) compare the directory structures in this directory tree to that in the original repo to see what is different (ignore files at this point). Nothing to do here, just get familiar with the differences. + +5. Type ``mv ../.git .`` to move the ``.git`` directory from the main repo to the cookiecutter repo. + +6. Create a new branch for all the changes, e.g., ``git checkout -b cookierelease``. + +7. Type ``cp -n -r ../src .`` to copy the source code from the main to the cookiecutter repo, without overwriting existing files in the destination. If there is no src directory, it will be something like ``cp -n -r ../diffpy ./src``. + +8. Type ``git status`` to see a list of files that have been (1) untracked, (2) deleted, (3) modified. Untracked files are in the cookiecutter but not in the original repo, deleted files are in the original but haven't been moved over, and modified files are in both but have been changed. + +9. Let's now copy over any documentation, similar to what we did with the src files. We want to copy over everything in the ``doc//source`` file from the old repo to the ``doc/source`` file in the new repo. + + 1. If you see this extra ``manual`` directory, run ``cp -n -r ../doc/manual/source/* ./doc/source``. + + 2. If files are moved to a different path, open the project in PyCharm and do a global search (ctrl + shift + f) for ``../`` or ``..`` and modify all relative path instances. + +10. Now we will work on correcting all the things that are wrong. + + 1. Add and commit each of the (1) untracked files to the git repo. These files are in the cookiecutter repo but not in the main repo, so can simply be "git added". Do it one (or a few) at a time to make it easier to rewind by having multiple commits. + + 2. Make a PR of your ``cookierelease`` branch by pushing your fork and opening a PR. + + 3. Files showing as (2) "deleted" upon git status are in the main repo but not in the cookiecutter repo. We took care of most of these by moving over the src tree, but let's do the rest now. Go down the list and for in the ``git status`` "delete" files type ``cp -n ..// ./``. Do not move files that we do not want. If you are unsure, feel free to confirm with Simon. + + 4. Files that have been (3) modified exist in both places and need to be merged **manually**. Do these one at a time. First open the file in PyCharm, then select ``Git|current file|show diff`` and the differences will show up. Select anything you want to inherit from the file in the main repo. For example, you want to copy useful information such as LICENSE and README files from the main repo to the cookiecutter repo. + + 5. Any files that we moved over from the old place, but put into a new location in the new repo, we need to delete them from git. For example, files that were in ``doc/manual/source/`` in the old repo but are not ``doc/source`` we correct by typing ``git add doc/manual/source``. + +11. Run pytest ``python -m pytest`` or ``pytest`` to make sure everything is working. There should be no errors if all tests passed previously when you were working on pre-commit. You may encounter deprecation warnings. There might be several possibilities: + + fixes separate from cookiecuttering. Remember to add it to Github issue. + + 2. Most ``pkg_resources`` deprecation warnings will be fixed by cookiecutter, but if you are in a diffpy package using unittests and see this warning you can fix them by replacing ``from pkg_resources import resource_filename`` with ``from importlib import resources`` and change ``path = resource_filename(__name__, p)`` to ``path = str(resources.files(__name__).joinpath(p))``. If you see ``collected 0 items no tests ran`` you might want to rename testing files as ``test_*.py``. Refer to the [migration guide](https://importlib-resources.readthedocs.io/en/latest/migration.html). + +.. _cookiecutter-workflow-api: + +3. API documentation workflow +----------------------------- + +This should be done only when the above steps are finished. + +When you see files with ``..automodule::`` within them, these are API documentation. However, these are not populated. We will populate them using our release scripts. + +1. Make sure you have our release scripts repository. Go to ``dev`` and run ``git clone https://github.com/Billingegroup/release-scripts.git``. + +2. Enter your cookiecutter package directory. For example, I would run ``cd ./diffpy.pdfmorph/diffpy.pdfmorph``. + +3. Build the package using ``python -m build``. You may have to install ``python-build`` first. + +4. Get the path of the package directory proper. In the case of ``diffpy.pdfmorph``, this is ``./src/diffpy/pdfmorph``. In general, for ``a.b.c``, this is ``./src/a/b/c``. + +5. Run the API script. This is done by running ``python ``. + + 1. If you have followed the steps above, the command is ``python ../../release-scripts/auto_api.py ./doc/source/api``. + +Make sure you build the documentation by going to ``/doc`` and running ``make html``. +The error "No module named" (``e.g. WARNING: autodoc: failed to import module 'tools' from module 'diffpy.pdfmorph'; the following exception was raised: No module named 'diffpy.utils'``) can be resolved by adding ``autodoc_mock_imports = []`` to your ``conf.py`` right under imports. This file is located in ``/doc/source/conf.py``. +In the case of ``PDFmorph``, this was done by adding ``autodoc_mock_imports = ["diffpy.utils",]``. + +Congratulations! You may now commit the changes made by ``auto_api.py`` (and yourself) and push this commit to the cloud! +Make a PR! It will be merged, trust! + +.. _cookiecutter-workflow-final: + +4. Final sign-off +----------------- + +#. For the ``cookierelease`` activity make a ``.rst`` file by copying ``TEMPLATE.rst`` in the news folder and under "fixed" put ``Repo structure modified to the new diffpy standard`` + +#. If a new Python version has been added under "added" add `Python 3.xx, 3,xx support`. If a previous version has been removed, under "fixed", add a new item `Python 3.xx, 3.xx, support`. + +#. Check the `README` and make sure that all parts have been filled in and all links resolve correctly. + +#. Run through the documentation online and do the same, fix grammar and make sure all links work. + +#. Follow the instructions on setting up GitHub pages here. + +.. _test-package-locally: + +Appendix 1. How to test your package locally +-------------------------------------------- + +Ensure your package has been cookiecuttered. We will use the ``diffpy.utils`` package as an example. In the package directory, follow these instructions: + +.. code-block:: bash + + # Create a new environment, specify the Python version and install packages + conda create -n diffpy_utils_env python=3.13 \ + --file requirements/test.txt \ + --file requirements/conda.txt \ + --file requirements/build.txt + + # Activate the environment + conda activate diffpy_utils_env + + # Install your package locally + # `--no-deps` to NOT install packages again from `requirements.pip.txt` + pip install -e . --no-deps + + # Run pytest locally + pytest + + # ... run example tutorials + +.. _build-documentation-locally: + +Appendix 2. How to build documentation locally +---------------------------------------------- + +Follow these steps sequentially: + +.. code-block:: bash + + cd doc + make html + open build/html/index.html + +To run as a single command: + +.. code-block:: bash + + cd doc && make html && open build/html/index.html && cd .. + +.. _build-documentation-preview-real-time: + +Real-time preview with Visual Studio Code +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Alternatively, you may render the Sphinx documentation by installing the `Esbonio `_ extension in VS Code. This will allow you to see the changes in real-time and increase productivity. + +.. _news-file-guide: + +Appendix 3. How to write ``.rst`` news file +----------------------------------------------------------------- + +We require that each PR includes a news item of ``.rst`` file under the ``news`` directory. + +Motivation and audience +^^^^^^^^^^^^^^^^^^^^^^^ + +``.rst`` files under the ``news`` directory are used to compile and update the ``CHANGELOG.rst`` file during releases. Hence, these news items are of interest to both developers and technical users looking for specific keywords. + +Guidelines for writing news items +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Do not remove ``news/TEMPLATE.rst``. Make a copy called ``.rst``. +- Do not modify other section headers in the rst file. Replace ``* `` only. See example news files in `Example 1 `_ and `Example 2 `_. +- Begin with "No news", "no news", or "no news added" for trivial changes with the following format: + +.. code-block:: text + + **Added:** + + * No news: + +Where to place the news item in ``.rst``? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- ``**Added:**`` includes features or functionality of interest to users and developers, such as support for a new Python version or the addition of a useful feature. +- ``**Changed:**`` includes modifications that affect end-users or developers, such as API changes or dependencies replaced. +- ``**Fixed:**`` includes bug fixes or refactoring. +- ``**Deprecated:**`` includes methods, classes, or workflows that are no longer supported in the future release. +- ``**Removed:**`` includes the opposite of the "Added" section, referring to features or functionality that have been removed. + +.. _codecov-token-setup: + +Appendix 4. Codecov token setup for the repository +-------------------------------------------------- + +For each PR, we use ``Codecov`` to report the test coverage percentage change as shown below. + +.. image:: ./img/codecov-pr.png + :alt: codecov-in-pr-comment + +To do so, the repository owner (Prof. Billinge) needs to provide a ``CODECOV_TOKEN`` at the repository level. This token is used to authenticate the GitHub CI with Codecov. Please follow the step-by-step guide below. + +1. Visit https://app.codecov.io/ + +2. Connect your repository or organization with Codecov by clicking ``Configure Codecov's GitHub app``, shown below: + +.. image:: ./img/codecov-configure.png + :alt: codecov-configure-github-project-button + +3. Scroll down, find your repository of interest, and click ``Configure``, shown below: + +.. image:: ./img/codecov-projects.png + :alt: codecov-list-github-projects + +4. Scroll down again, copy ``CODECOV_TOKEN``, shown below: + +.. image:: ./img/codecov-token.png + :alt: codecov-list-github-projects + +5. In your GitHub repository, go to ``Settings``, then click ``Actions`` under the ``Secrets and Variables`` tab. + +6. Click ``New repository secret``. + +7. Paste the token value and name it as ``CODECOV_TOKEN`` secret as shown below: + +.. image:: ./img/codecov-github.png + :alt: codecov-list-github-projects + +8. Done. The Codecov token is now set up for the repository. A comment will be generated on each PR with the Codecov status automatically. + +.. _pre-commit-github-repo-setup: + +Appendix 5. How to configure pre-commit CI via GitHub Apps +---------------------------------------------------------- + +``Pre-commit CI`` is available as a GitHub app that executes pre-commit hooks in each pull request, as shown in the image below. While it is recommended to run ``precommit run --all-files`` locally before making a PR, this GitHub app will automatically attempt to lint code and format docstrings according to the hooks provided in ``.pre-commit-config.yaml``. If all passes, it will give you a green checkmark as shown below. + +.. image:: ./img/precommit-PR.png + :alt: pre-commit-PR-automatic-check + +To configure ``pre-commit CI``, follow the simple steps below: + +1. Visit https://github.com/apps/pre-commit-ci and click "Configure". +2. Select the repository(s). +3. Done! + +Appendix 6. How to setup GitHub Pages for your package +------------------------------------------------------ + +You have API doc built and tested locally. Now you want to deploy your doc via ``https://org-name/github.io/repo-name`` i.e., https://diffpy.github.io/diffpy.utils using GitHub Pages. + +Go to `Settings` in your repository. diff --git a/_sources/cookiecutter_guide.rst.txt b/_sources/cookiecutter_guide.rst.txt deleted file mode 100644 index 41e1e4f..0000000 --- a/_sources/cookiecutter_guide.rst.txt +++ /dev/null @@ -1,178 +0,0 @@ -:tocdepth: -1 - -.. index:: cookiecutter_guide - -================================== -How to migrate an existing package -================================== - -The following creates and activates a new environment named ``cookiecutter_env`` :: - - conda create -n cookiecutter_env python=3.13 - -Activate the environment :: - - conda activate cookiecutter_env - -Install packages :: - - pip install cookiecutter black pre-commit - -Overview --------- - -The cookiecutting is divided into three main sections: - -1. Pre-commit workflow: Use ``black`` and ``pre-commit`` to standardize line lengths and remove flake8 errors. - -2. Cookiecutting workflow: Use the Cookiecutter project setup and move files from an older to a new structure with Pytest passing. - -3. API documentation build workflow: Use a script to build and host API documentation. - -WARNINGS --------- - -- Do not delete/remove any files before confirming that it is absolutely not necessary. Create an issue or contact the maintainer for assistance. - -- When copying over documentation files, make sure you include any additional package-specific information that may be in those files. - -- For instance, there may be a more verbose description of what the package does, or tutorial/example/utility files. - - -1. Pre-commit workflow ----------------------- - -1. In your ``dev`` folder, fork and clone the package - -2. ``cd`` into the top level directory of that project. - -3. Type ``git pull upstream main`` to sync with the main branch. - -4. Double check that no bug-fix etc. pull-requests are waiting to be merged. Check with Simon if not sure. - -5. Create a new branch called ``black``. - -6. Create ``pyproject.toml``. Copy and paste the ``[tools.black]`` and ``[tool.codespell]`` sections from ``pyproject.toml`` in the ``{{ cookiecutter.repo_name }}`` folder path. - -7. Run ``black src`` (note: some of the older packages do not have an ``src`` directory, so you may have to run black on a different directory). - -8. Commit the automatic changes by ``black``. - -9. Run ``black .`` and create a PR into ``main``. Follow the group's GitHub workflow tutorial on GitLab. - -10. After the ``black`` branch has been merged, run pytest or unit tests to ensure all tests pass locally. If the code is failing, please consult with Simon before further proceeding. - -11. Type ``git checkout main && git pull upstream main`` and create a new branch called ``precommit``. - -12. Copy and paste the ``.flake8`` and ``.pre-commit-config.yaml`` files from ``{{ cookiecutter.repo_name }}`` to the top directory level. Cross-check with https://github.com/diffpy/diffpy.utils. - -13. Run ``pre-commit run --all-files``. Fix any spelling suggestions from Codespell. To ignore a specific word or line, add it under ``.codespell/ignore_words.txt`` or ``.codespell/ignore_lines.txt``. To ignore specific file types, add the file extensions i.g. ``*.gr`` in ``skip = line`` under ``[tool.codespell]`` in ``pyproject.toml``. Include explanations for each addition. - -14. Create a PR to ``main``. Mention in the PR that you need to address flake8 errors. - -15. After the ``precommit`` branch has been merged, sync with ``main`` in Step 11, create a new branch called ``flake8`` - -16. Fix flake8 errors manually: - - - Tip 1: Start with easier error types to fix, such as line-lenghts and "module imported not used", etc. - - - Tip 2: Submit periodic commits within a single PR. - - - Tip 3: Create multiple PRs, each containing a specific theme (e.g., "Fix docstring line-length flake8 errors" using ``flake8-length`` branch, etc.) to reduce cognitive overload for the reviewer (Simon). - - - Tip 4: Don't hesitate to reach out to group members who have contributed to this repository. They probably have seen those errors before! - -17. Only proceed to the next section after addressing all PRs relevant to the pre-commit workflow. - -2. Cookiecutter workflow ------------------------- - -1. Type ``cookiecutter https://github.com/billingegroup/cookiecutter`` inside the package directory - -2. . Answer the questions as the following -- note that (default) means to hit enter without modifying anything: - - 1. (May occur if it is not the first time you have installed) Is it okay to delete: (y) - - 2. github_org: diffpy (if diffpy project) or billingegroup - - 3. keywords: current keywords in the ``setup.py`` or ``pyproject.toml`` in comma-separated string format, e.g., ``diffpy, pdf, something, something else`` (no quotes around it) - - 4. project_name: - - 5. package_dist_name: (default) - - 6. package_dir_name: (default) - - 7. repo_name: (default) - - 8. minimum_python_version: (default -- this is 3.11) - - 9. maximum_python_version: (default -- this is 3.13) - - 10. have_c_code: no (in general, but if there are C++ extensions, this will be yes) - - 11. Is a GUI application, run 'HEADLESS' tests (default: false): (default). If it is a GUI package, change this to ``true``. - - 12. Enter value for workflow 'VERSION' (default: v0): (default). Version of the workflow to use. - -4. cd into the new ``diffpy./`` directory (e.g., in our example ``pwd`` would return ``~/dev/diffpy.pdfmorph/diffpy.pdfmorph``) (we will refer to the nested directory as the "**cookiecutter**" directory and ``~/dev/diffpy.pdfmorph/`` as the "**main**" directory). - -5. Type ``ls -als`` (if you have the alias, this is ``ll``) compare the directory structures in this directory tree to that in the original repo to see what is different (ignore files at this point). Nothing to do here, just get familiar with the differences. - -6. Type ``mv ../.git .`` to move the ``.git`` directory from the main repo to the cookiecutter repo. - -7. Create a new branch for all the changes, e.g., ``git checkout -b cookierelease``. - -8. Type ``cp -n -r ../src .`` to copy the source code from the main to the cookiecutter repo, without overwriting exiting files in the destination. If there is no src directory, it will be something like ``cp -n -r ../diffpy ./src``. - -9. Type ``git status`` to see a list of files that have been (1) untracked, (2) deleted, (3) modified. Untracked files are in the cookiecutter but not in the original repo, deleted files are in the original but haven't been moved over, and modified files are in both but have been changed. - -10. Let's now copy over any documentation, similar to what we did with the src files. We want to copy over everything in the ``doc//source`` file from the old repo to the ``doc/source`` file in the new repo. - - 1. If you see this extra ``manual`` directory, run ``cp -n -r ../doc/manual/source/* ./doc/source``. - - 2. If files are moved to a different path, open the project in PyCharm and do a global search (ctrl + shift + f) for ``../`` or ``..`` and modify all relative path instances. - -11. Now we will work on correcting all the things that are wrong. - - 1. Add and commit each of the (1) untracked files to the git repo. These files are in the cookiecutter repo but not in the main repo, so can simply be "git added". Do it one (or a few) at a time to make it easier to rewind by having multiple commits. - - 2. Make a PR of your ``cookierelease`` branch by pushing your fork and opening a PR. - - 3. Files showing as (2) "deleted" upon git status are in the main repo but not in the cookiecutter repo. We took care of most these by moving over the src tree, but let's do the rest now. Go down the list and for in the ``git status`` "delete" files type ``cp -n ..// ./``. Do not move files that we do not want. If you are unsure, feel free to confirm with Simon. - - 4. Files that have been (3) modified exist in both places and need to be merged **manually**. Do these one at a time. First open the file in pycharm, then select ``Git|current file|show diff`` and the differences will show up. Select anything you want to inherit from the file in the main repo. For example, you want to copy useful information such as LICENSE and README files from the main repo to the cookiecutter repo. - - 5. Any files that we moved over from the old place, but put into a new location in the new repo, we need to delete them from git. For example, files that were in ``doc/manual/source/`` in the old repo but are not ``doc/source`` we correct by typing ``git add doc/manual/source``. - -12. Run pytest ``python -m pytest`` to make sure everything is working. There should be no errors if all tests passed previously when you were working on pre-commit. You may encounter deprecation warnings. There might be several possibilities: - - 1. If you see numpy deprecation warnings, we won't clean up these deprecations now. Pin numpy to 1.x for now to get tests to pass. Do code fixes separate from cookiecuttering. Remember to add it to Github issue. - - 2. Most ``pkg_resources`` deprecation warnings will be fixed by cookiecutter, but if you are in a diffpy package using unittests and see this warning you can fix them by replace ``from pkg_resources import resource_filename`` with ``from importlib import resources`` and change ``path = resource_filename(__name__, p)`` to ``path = str(resources.files(__name__).joinpath(p))``. If you see ``collected 0 items no tests ran`` you might want to rename testing files as ``test_*.py``. Refer to the [migration guide](https://importlib-resources.readthedocs.io/en/latest/migration.html). - -3. API documentation workflow ------------------------------ - -This should be done only when the above steps are finished. - -When you see files with ``..automodule::`` within them, these are API documentation. However, these are not populated. We will populate them using our release scripts. - -1. Make sure you have our release scripts repository. Go to ``dev`` and running ``git clone https://github.com/Billingegroup/release-scripts.git``. - -2. Enter your cookiecutter package directory. For example, I would run ``cd ./diffpy.pdfmorph/diffpy.pdfmorph``. - -3. Build the package using ``python -m build``. You may have to install ``python-build`` first. - -4. Get the path of the package directory proper. In the case of ``diffpy.pdfmorph``, this is ``./src/diffpy/pdfmorph``. In general, for ``a.b.c``, this is ``./src/a/b/c``. - -5. Run the API script. This is done by running ``python ``. - - 1. If you have followed the steps above, the command is ``python ../../release-scripts/auto_api.py ./doc/source/api``. - -Make sure you build the documentation by going to ``/doc`` and running ``make html``. -The error "No module named" (``e.g. WARNING: autodoc: failed to import module 'tools' from module 'diffpy.pdfmorph'; the following exception was raised: No module named 'diffpy.utils'``) can be resolved by adding ``autodoc_mock_imports = []`` to your ``conf.py`` right under imports. This file is located in ``/doc/source/conf.py``. -In the case of ``PDFmorph``, this was done by adding ``autodoc_mock_imports = ["diffpy.utils",]``. - -Congratulations! You may now commit the changes made by ``auto_api.py`` (and yourself) and push this commit to the cloud! -Make a PR! It will be merged, trust! diff --git a/_sources/frequently-asked-questions.rst.txt b/_sources/frequently-asked-questions.rst.txt new file mode 100644 index 0000000..0bddc8d --- /dev/null +++ b/_sources/frequently-asked-questions.rst.txt @@ -0,0 +1,345 @@ +:tocdepth: -1 + +.. index:: frequently-asked-questions + +.. _frequently-asked-questions: + +================================ +Frequently asked questions (FAQ) +================================ + +Here, you will learn how to customize the ``bg-cookiecutter`` template for your own project, such as setting the line-width and including/excluding files for PyPI distribution. We also provide design decisions for the current setup of the ``bg-cookiecutter`` template. + +Formatting +---------- + +How do I modify line-width limits? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Three files need to be modified: + +1. In ``.isort.cfg``, modify ``line_length`` +2. In ``.flake8``, modify ``max-line-length`` +3. In ``pyproject.toml``, modify ``line-length`` under ``[tool.black]``. + +Pre-commit +---------- + +.. codespell-add-word: + +How do I ignore words/lines/files in automatic spelling checks in pre-commit? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To ignore a word, add it to ``.codespell/ignore_words.txt``. + +To ignore a specific line, add it to ``.codespell/ignore_lines.txt``. See the example below: + +.. code-block:: text + + ;; src/translation.py + ;; The following single-line comment is written in German. + # Hallo Welt + +To ignore a specific file extension, add ``*.ext`` to the ``skip`` section under ``[tool.codespell]`` in ``pyproject.toml``. For example, to ignore ``.cif`` and ``.dat`` files, use ``skip = "*.cif,*.dat"``. + +Release +------- + +.. _release_authority: + +How can I change who is authorized to release a package? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In ``.github/workflows/build-wheel-release-upload.yml``, modify ``github_admin_username`` to the desired GitHub username. This username will be able to authorize the release by pushing the tag as instructed :ref:`here `. + +How is the package version set and retrieved? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In ``pyproject.toml``, the ``setuptools-git-versioning`` tool is used to dynamically retrieve the version based on the latest tag in the repository. The latest tag is pushed during the release workflow. The dynamically parsed version is globally accessible, via ``.__version__``. + +How do I include/exclude files in PyPI release? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``MANIFEST.in`` file is used to control which files are included in the source distribution. Try running ``python -m build`` and see the content under the ``dist`` folder generated. + +To include all files under a folder, use ``graft``: + +.. code-block:: text + + graft src + graft tests + +To include specific file(s), use ``include``: + +.. code-block:: text + + include AUTHORS.txt LICENSE*.txt README.rst + +To exclude files globally, use ``globally-exclude``: + +.. code-block:: text + + global-exclude *.py[cod] # Exclude all .pyc, .pyo, and .pyd files. + global-exclude .DS_Store # Exclude Mac filesystem artifacts. + global-exclude __pycache__ # Exclude Python cache directories. + global-exclude .git* # Exclude git files and directories. + +Why have we decided to include test files in the PyPI source distribution? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We decided to include test files in the PyPI source distribution to facilitate unit testing with a newly built Conda package. + +The conda-forge CI uses the source code distributed via PyPI to build a Conda package. After building the package, we want to run pytest to ensure all unit tests pass before release. Therefore, test files must be included in the source code. In contrast, no documentation is distributed with the package, as it is already accessible from the GitHub repository and does not serve a practical purpose in the distribution package itself. + +Documentation +------------- + +How can I preview documentation in real-time? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You may use Visual Studio Code. Please refer to the following section :ref:`here `. + +How do I re-deploy online documentation without another release? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Visit the following URL of your package: ``https://github.com///actions/workflows/publish-docs-on-release.yml`` i.e., https://github.com/diffpy/diffpy.utils/actions/workflows/publish-docs-on-release.yml. + +Click ``Run workflow`` and select the ``main`` branch. Your online documentation will be updated with the latest changes without a new release. + +conda-forge +----------- + +How do I add a new admin to the conda-forge feedstock? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Please refer to the admin section in the conda-forge release guide :ref:`here `. + +How do I do pre-release for conda-forge? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Please read our pre-release section in the conda-forge release guide :ref:`here `. + +GitHub Actions +-------------- + +How do I set different Python versions for GitHub CI? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default is Python 3.13 for ``_tests-on-pr.yml`` and ``_publish-docs-on-release.yml``. Python 3.11, 3.12, and 3.13 are used for ``_matrix-and-codecov-on-merge-to-main.yml``. To override the default, modify the three ``.yml`` files above in ``.github/workflows/`` as shown below: + +1. Add ``python_version`` in ``.github/workflows/tests-on-pr.yml``: + +.. code-block:: yaml + + jobs: + tests-on-pr: + uses: Billingegroup/release-scripts/.github/workflows/_tests-on-pr.yml@v0 + with: + project: package-name + c_extension: false + headless: false + python_version: 3.12 + secrets: + CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} + +2. Add ``python_version`` in ``.github/workflows/_publish-docs-on-release.yml``: + +.. code-block:: yaml + + jobs: + docs: + uses: Billingegroup/release-scripts/.github/workflows/_tests-on-pr.yml@v0 + with: + project: package-name + c_extension: false + headless: false + python_version: 3.12 + +3. Add ``python_versions`` in ``.github/workflows/_matrix-and-codecov-on-merge-to-main.yml``: + +.. code-block:: yaml + + jobs: + matrix-coverage: + uses: Billingegroup/release-scripts/.github/workflows/_matrix-and-codecov-on-merge-to-main.yml@v0 + with: + ... + python_versions: "3.11, 3.12" + +What is the difference between ``pull_request`` and ``pull_request_target``? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For the current GitHub CI for checking a news item, ``pull_request_target`` is used instead of ``pull_request`` as shown below: + +.. code-block:: yaml + + name: Check News Item + + on: + pull_request_target: + branches: + - main + +- ``pull_request``: This event configures the ``GITHUB_TOKEN`` with read-only permissions by default, especially when triggered by forks. +- ``pull_request_target``: This event grants the ``GITHUB_TOKEN`` write permissions, enabling it to perform actions that modify the repository, such as posting comments, updating pull request statuses, or merging code. The news CI creates a comment when an additional news ``.rst`` is not found under the ``news`` folder. Hence, ``pull_request_target`` is used. + +Another key difference is that with ``pull_request_target``, the ``.yml`` file **must already be merged** in the base branch at the time the pull request is opened or updated. For more, please refer to `GitHub docs `_. + +Dependency management +--------------------- + +Why are both pip.txt and conda.txt provided? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Our preferred choice for installing the cookiecuttered package is as a Conda package, as outlined in the template ``README.rst`` file. With Conda, the end user can install all associated dependencies by running ``conda create --name new_env ``. Additionally, the environment is tested via conda-forge CI before the Conda package is released, which helps ensure the package's compatibility with its dependencies. Hence, we list conda package dependencies in ``conda.txt``. + +However, we also want to allow users to install the package via ``pip``. To support this, we provide a separate file for pip dependencies, ``pip.txt``. In most cases, the dependencies listed in ``conda.txt`` and ``pip.txt`` will be identical. However, there can be exceptions. For example, ``matplotlib-base`` is preferred for Conda installations, while ``matplotlib`` is used for pip installations. + +GitHub workflow +--------------- + +I am new to GitHub. Why do we use Git/GitHub? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +GitHub allows multiple contributors to work on a software project simultaneously under an organization like ``Billingegroup`` or ``diffpy``. There are two primary needs. First, we want to ensure that any changes under this organization are reviewed by the organization's project owner. Second, we want to ensure we add new changes from the latest version of the code, particularly when working with multiple contributors across different time zones. Hence, we use GitHub to serve the needs with a specific workflow below. Please see below for an overview of the GitHub workflow. + +.. _github-workflow-overview: + +What is the general the workflow? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Since cookiecutting requires a basic understanding of GitHub's workflow, we will provide you with a brief overview and how to set up your repository. + +First, if you are working on a package from an organization like ``github.com/diffpy`` or ``github.com/Billingegroup``, you first copy the repository of the organization to your GitHub user account. This process is called ``forking``. + +Then, you will download the forked repository in your GitHub account to your local machine. This process is called ``cloning``. + +In the cloned repository on your local machine, you will make edits. You want to first add a description for the changes by "committing" with a message describing the changes. Then you will upload these changes to the ``forked`` repository in your account. This process of updating code from the local computer to the repository hosted by GitHub is called ``pushing``. + +From the forked repository, you then want to upload changes to the repository under ``github.com/Billingegroup/cookiecutter``, for example. This process is done through a process called ``pull request``. The Project Owner reviews this pull request and merges it into the Billinge group's repository. If you are the contributor as well as the Project Owner, you would be the one who reviews your own code and merges your changes. + +I have a general understanding of fork, clone, commit, push, and pull request. How do I set up my repository for cookiecutting? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Please be familiar with the terminology such as "fork", "clone", "push", and "pull request" :ref:`above `. + +You may fork the repository using the "Fork" button on the top right corner of the repository page. This will copy the repository to your GitHub account. e.g., ``github.com/Billingegroup/cookiecutter`` to ``github.com/sbillinge/cookiecutter``. + +Then download the forked repository under your account to the local machine by cloning: + +.. code-block:: bash + + git clone https://github.com// + +Now, you also want to link with the repository of the organization by adding the URL. Recall, we want to make changes from the latest state of the source code. + +.. code-block:: bash + + git remote add upstream https://github.com// + +.. note:: + + What is ``upstream``? The repository that you forked from, e.g. ``Billingegroup/cookiecutting`` is referred to as the ``upstream`` repository. + +Verify that you have the ``upstream`` URL set up as the organization. + +.. code-block:: bash + + git remote -v + +Notice that you also have ``origin`` with an URL linking to your forked repository under your account. This is another GitHub jargon that refers to your forked repository. + +.. note:: + + What is ``remote``? The term ``remote`` is the opposite of ``local``. In other words, ``remote`` refers to the repository that is hosted by GitHub. e.g., ``github.com/Billingegroup/cookiecutter`` or ``github.com/sbillinge``. + +Do you have a general summary of each term used in the GitHub workflow? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +:fork: The process of copying a repository from an organization to your GitHub account. e.g., ``github.com/Billingegroup/cookiecutter`` to ``github.com/sbillinge/cookiecutter``. + +:upstream: The repository of the original source code. e.g., ``github.com/Billingegroup/cookiecutter``. + +:origin: The forked repository under your account. e.g., ``github.com/sbillinge/cookiecutter``. + +:remote: The repository that is hosted by GitHub. e.g., ``github.com/Billingegroup/cookiecutter`` or ``github.com/sbillinge/cookiecutter``. + +:branch: The branch serves as a folder that contains the files of the repository. The ``main`` branch is the branch that is used for the final version of the code. Many branches can be created for different features or bug fixes that are later merged into the ``main`` branch. + +:git clone: The process of locally downloading a repository from GitHub (``remote``) to your local machine. + +:git push: The process of updating code from the local computer to the GitHub remote repository. Push can be made to the ``origin`` or ``upstream`` repository. But, in our workflow, we push to the ``origin`` repository, and then we create a pull request to merge the changes from ``origin`` to the ``upstream`` repository. + +:git commit: The process of adding a description for the changes made in the files that are ready to be pushed. + +:git add: The process of selecting files to be included within a single commit. + +I have cloned and added ``upstream``. What is the next step? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We want to first sync our local folder with the ``upstream`` repository. This process is called ``pulling``. + +.. code-block:: bash + + git checkout main + git pull upstream main + +Above, we checkout the ``main`` branch of your cloned folder. We then download all the latest changes from the ``upstream`` repository. Recall that a GitHub repository is contributed by multiple contributors. Hence, we want to ensure that we are working with the latest version of the code in the ``main`` branch. + +Once we are fully synced with the ``upstream`` repository, we can now start making changes to the code. + +Instead of directly working in the ``main`` branch of your cloned repository, you will create a copy of ``main`` by "branching" it from ``main``. Think of a tree. You can name it anything you want like ``docs-faq``, etc. + +.. code-block:: bash + + git checkout -b docs-faq + +The above command not only creates a new branch but also switches to the new branch. You can verify that you are in the new branch by running: + +.. code-block:: bash + + git branch + +Of course, you can always switch back to the ``main`` branch by using ``git checkout main``. + +Now, you are ready to make changes to the code in the branch. If you have a README file in your project, try to modify it. Once you are done, you want to add the changes to a hidden folder called ``.git``. This process is called ``staging``. + +.. code-block:: bash + + git add README.rst + +Then, now you want to commit the changes with a message describing the changes. + +.. code-block:: bash + + git commit -m "docs: added a FAQ section in the README" + +Now, you want to push the changes to the ``origin`` repository under your account. Recall ``origin`` refers to the forked repository under your account hosted by GitHub. + +.. code-block:: bash + + git push --set-upstream origin docs-FAQ + +Go to your forked repository under your account on GitHub. You will see a green button that says "Compare & pull request". Click on it. You will see the changes you made in the branch. Click on "Create pull request". Add a description of the changes you made. Click on "Create pull request". + +The reviewer will review the changes and merge them into the ``upstream`` repository. You have successfully made your first contribution to the organization's repository. + +I still need to make another pull request. How do I do that? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, you want to make another pull request. You want to make sure that you are working with the latest version of the code in the ``main`` branch. + +.. code-block:: bash + + git checkout main + git pull upstream main + +The command above will sync your local folder with the ``upstream`` repository. It should download the changes made by other contributors as well as the recent commit you made in the ``docs-FAQ`` branch, for example. + +Again, you checkout a new branch from the ``main`` branch. You can name it anything you want, e.g. ``docs-typo``. + +.. code-block:: bash + + git checkout -b docs-typo + +You repeat the process of git add, commit, push to your ``origin`` (your forked repository) and then make a PR to the ``upstream`` repository (the organization's repository). diff --git a/_sources/getting-started.rst.txt b/_sources/getting-started.rst.txt new file mode 100644 index 0000000..6553bd5 --- /dev/null +++ b/_sources/getting-started.rst.txt @@ -0,0 +1,25 @@ +:tocdepth: -1 + +.. index:: getting-started + +.. _getting-started: + +=============== +Getting started +=============== + +Cookiecutting refers to the process of standardizing a Python package structure using the project template. + +Are you here for cookiecutting? +------------------------------- + +1. You have an existing Python package. Do you want to standardize your package with ``bg-cookiecutter``? Please follow the full instructions :ref:`here `. + +2. You already have a bg-cookiecuttered package. Do you want to use the latest version of ``bg-cookiecutter`` to re-cookiecut your package? Please follow the instructions in :ref:`here `. + +3. Do you want to start a new Python repository with ``bg-cookiecutter``? Start from the Cookiecutter workflow section :ref:`here `. + +Are you here for release? +------------------------- + +Do you want to release your cookiecuttered package to ``GitHub``, ``PyPI``, and ``conda-forge`` by creating a tag to your GitHub repository? Start from :ref:`here `. diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt index 53251fa..fe89617 100644 --- a/_sources/index.rst.txt +++ b/_sources/index.rst.txt @@ -4,19 +4,82 @@ .. |title| replace:: bg-cookiecutter documentation -bg-cookiecutter - Python package standard in the Billinge Research Group - | Software version |release|. | Last updated |today|. +Welcome to Billinge Group's Python cookiecutter documentation! + +Statement of need +----------------- + +Your research impact can be amplified by distributing your contributions through well-documented software packages. However, creating these packages requires time dedicated to software infrastructure and ongoing maintenance. ``bg-cookiecutter`` is a Python package standard and generator that helps you launch new scientific code or migrate existing ones. + +``bg-cookiecutter`` provides research-quality documents suitable for open-source development and supports Python versions compatible with the `SPEC0 `_ proposal through automated workflows. Additionally, it offers a streamlined process for building and distributing your scientific software, ensuring reproducibility across all major operating systems. With ``bg-cookiecutter``, you can focus on developing your scientific code while ensuring it reaches a broader audience. + +What are the useful features? +----------------------------- + +To help increase your research impact, ``bg-cookiecutter`` offers the following features: + +- Automated `PEP8 `_ and `PEP256 `_ standard checks. +- Automated PyPI/GitHub release, testing, documentation, and CHANGELOG updates. +- Streamlined package release workflow with a checklist. +- Latest Python version support compatible with `SPEC0 `_. +- Rich README template containing badges, installation, support, and contribution guide. +- Automatic spelling check, linting for .json, .yml, and .md files. + +For more technical details, it includes: + +- Pull requests with coverage reports using ``Codecov`` and checks with ``pre-commit CI``. +- Namespace package support, e.g., ``import diffpy.utils``. +- conda-package ``meta.yaml`` generation with a template. +- Support for non-pure Python package releases with ``cibuildwheel``. +- Support for headless GitHub CI testing for GUI applications. +- Reusable GitHub Actions workflows located in `Billingegroup/release-scripts `_. + +How do I get started? +--------------------- + +Please visit the :ref:`Getting started ` page to learn how to navigate the documentation! + +How do I receive support? +------------------------- + +If you have any questions or have trouble, please read the :ref:`Frequently asked questions (FAQ) ` section to see if your questions have already been answered. If there aren't answers available, please create an issue. The team will reach out. + +How can I contribute? +--------------------- + +Do you have any new features? Please make an issue via the GitHub issue tracker for further discussions. For a minor typo or grammatically incorrect sentence, please make a pull request. Before making a PR, please run ``pre-commit run --all-files`` to ensure the code is formatted. + +Who are using ``bg-cookiecutter``? +---------------------------------- + +The full list of packages is as follows: + +- `diffpy.pdffit2 `_ +- `diffpy.fourigui `_ +- `diffpy.pdfgui `_ +- `diffpy.utils `_ +- `diffpy.structure `_ +- `diffpy.labpdfproc `_ +- `diffpy.pdfmorph `_ +- `diffpy.snmf `_ +- `diffpy.srmise `_ +- `regolith `_ +- `bg-mpl-stylesheets `_ + ======= Authors ======= -bg-cookiecutter is developed by Billinge Group -and its community contributors. +- Sangjoon Lee (sl5400@columbia.edu) +- Andrew Yang (ay2546@columbia.edu) +- Simon Billinge (sb2896@columbia.edu) + +bg-cookiecutter is developed by Billinge Group and its community contributors. -For a detailed list of contributors see +For a detailed list of contributors, see https://github.com/Billingegroup/cookiecutter/graphs/contributors. ================ @@ -31,9 +94,11 @@ Table of contents .. toctree:: :maxdepth: 2 - cookiecutter_guide - release_guide - conda_forge_guide + getting-started + cookiecutter-guide + release-guide + conda-forge-guide + frequently-asked-questions license release diff --git a/_sources/release-guide.rst.txt b/_sources/release-guide.rst.txt new file mode 100644 index 0000000..2ba52b1 --- /dev/null +++ b/_sources/release-guide.rst.txt @@ -0,0 +1,140 @@ +:tocdepth: -1 + +.. index:: release-guide + +.. _release-guide: + +=============================== +How to release Python package +=============================== + +Overview +~~~~~~~~ + +This guide is written for those who have just cookiecuttered. + +If you already have a conda package and if you are interested in just release a new version, read the instruction here. + +PyPI/GitHub release +~~~~~~~~~~~~~~~~~~~~ + +.. _release-instructions-contributor: + +Make sure you have your package cookiecuttered. Otherwise, please start from the Getting started page :ref:`here ` + +#. In the repository, create an issue on GitHub with the "Release" option as shown below: + + .. image:: ./img/add-personal-access-token.png + :alt: add-personal-access-token + :width: 600px + +#. Check off all items in the first checklist for PyPI/GitHub release. + +Instructions for Project Owner for release +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. _release-instructions-project-owner: + +#. Review the checklist prepared by the contributor in the issue. + +#. Ensure ``PYPI_TOKEN`` and ``PAT_TOKEN`` are configured at the org level. See Appendix :ref:`1 `, :ref:`2 `, respectively. + +#. Check the ``github_admin_username`` section in ``.github/workflows/build-wheel-release-upload.yml`` is that of the project owner. + +#. In your terminal, run ``git checkout main && git pull upstream main`` to sync with the main branch. + +#. Run the following:: + + # For pre-release, use *.*.*rc* i.g. 1.0.0rc0 + # For release, use *.*.* i.g. 1.0.0 + git tag + git push upstream + +#. Done! Once the tag is pushed, visit the ``Actions`` tab in the repository to monitor the CI progress. + +#. Comment ``GitHub/PyPI release done!`` or ``GitHub/PyPI pre-release done!`` in the release GitHub issue. + +#. The contributor will make a PR into the conda-forge feedstock. Review and approve the PR. + +#. Wait for the contributor to test the released package. The issue will be closed by the contributor. + +.. _pypi-token-setup: + +Appendix 1. Setup ``PYPI_TOKEN`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Generate a PyPI API token from ``pypi.org``: + +#. Visit https://pypi.org/manage/account/ and log in. + +#. Scroll down to the ``API tokens`` section and click ``Add API token``. + +#. Set the ``Token name`` to ``PYPI_TOKEN``. + +#. Choose the appropriate ``Scope`` for the token. + +#. Click ``Create token`` and copy the generated token. + +Add the generated token to GitHub: + +#. Navigate to the ``Settings`` page of the org (or repository). + +#. Click the ``Actions`` tab under ``Secrets and variables``. + +#. Click ``New org secret``, name it ``PYPI_TOKEN``, and paste the token value. + +#. Done! + +.. image:: ./img/add-pypi-secret.png + :alt: add-pypi-secret + :width: 600px + +.. _pat-token-setup: + +Appendix 2. Setup ``PAT_TOKEN`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The GitHub workflow needs to link with this privilege through a personal access token (PAT) of the project owner. + +1. Visit https://github.com/settings/tokens + +2. Click ``Generate new token`` and choose the classic option. + +3. Under ``Note``, write, "GitHub CI release" + +4. Set the Expiration date of the token. + +5. Under ``Select scopes``, check ``repo`` and ``user``. + +6. Scroll down, click ``Generate token``. + +7. Done! + +.. image:: ./img/add-personal-access-token.png + :alt: add-personal-access-token + :width: 600px + +Copy and paste the ``PAT_TOKEN`` to your GitHub organization: + +1. Visit ``Settings`` in the organization. + +2. Click the ``Actions`` tab under ``Secrets and variables``. + +3. Click ``New organization secret`` and add a new secret and name it as ``PAT_TOKEN``. + +4. Done! + +.. _gh-pages-setup: + +Appendix 3. How to host your package documentation online +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You have API doc built and tested locally. Now you want to deploy your doc via i.e., ``https://diffpy.github.io/diffpy.utils`` using GitHub Pages. + +#. Go to your ``Settings`` page in your repository and and click ``pages`` under ``Code and automation``. +#. Click ``Deploy from a branch`` under ``Source``. Below, choose ``gh-pages`` branch and ``/(root)`` and click ``Save``. + +.. image:: ./img/github-pages.png + :alt: setup-github-pages-from-branch + +Done! Wait a few minutes and visit your GitHub Pages URL! diff --git a/_sources/release_guide.rst.txt b/_sources/release_guide.rst.txt deleted file mode 100644 index dcff557..0000000 --- a/_sources/release_guide.rst.txt +++ /dev/null @@ -1,125 +0,0 @@ -:tocdepth: -1 - -.. index:: release_guide - -=============================== -How to release a Python package -=============================== - - -FAQ for PyPI/GitHub release -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Q1. No conda-forge feedstock yet? Create one by following the instructions :ref:`here `. - -Q2. Not a feedstock admin? Create an issue with the title/comment: ``@conda-forge-admin, please add user @username``. Example `issue `_. - -Q3. Interested in pre-release? Please follow the instructions under the ``conda-forge: pre-release`` section :ref:`here `_. - - - -Instructions for GitHub repository contributors -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When you are ready for PyPI and GitHub release, please follow these steps: - -1. In the repository, create an issue on GitHub with the "Release" option. - -2. Check off all items in the first checklist for PyPI/GitHub release. - -3. After PyPI/GitHub release, create a PR into the conda-forge feedstock. While doing so, check off items in the second checklist. See Appendix 1. - -4. After conda release, complete the third checklist and close the issue. - - -Instructions for Prof. Billinge for release -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Review the checklist prepared by the contributor in the issue. - -2. Ensure ``PYPI_TOKEN`` and ``PAT_TOKEN`` are configured at the org level. See Appendix :ref:`1 `, :ref:`2 `, respectively. - -3. In your terminal, run ``git checkout main && git pull upstream main`` to sync with the main branch. - -4. Run the following:: - - # For pre-release, use *.*.*rc* i.g. 1.0.0rc0 - # For release, use *.*.* i.g. 1.0.0 - git tag - git push upstream - -5. Done! Once the tag is pushed, visit the ``Actions`` tab in the repository to monitor the CI progress. - -Post GitHub/PyPI release for conda-forge release: - -5. Comment ``GitHub/PyPI release done!`` or ``GitHub/PyPI pre-release done!`` in the issue. - -6. The contributor will make a PR into the conda-forge feedstock. Review and approve the PR. - -7. Wait for the contributor to test the released package. The issue will be closed by the contributor. - -.. _appendix_pypi_token: - -Appendix 1. Setup ``PYPI_TOKEN`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Generate a PyPI API token from ``pypi.org``: - -1. Visit https://pypi.org/manage/account/ and log in. - -2. Scroll down to the ``API tokens`` section and click ``Add API token``. - -3. Set the ``Token name`` to ``PYPI_TOKEN``. - -4. Choose the appropriate ``Scope`` for the token. - -5. Click ``Create token`` and copy the generated token. - -Add the generated token to GitHub: - -1. Navigate to the ``Settings`` page of the org (or repository). - -2. Click the ``Actions`` tab under ``Secrets and variables``. - -3. Click ``New org secret``, name it ``PYPI_TOKEN``, and paste the token value. - -4. Done! - -.. image:: ./img/add-pypi-secret.png - :alt: add-pypi-secret - :width: 600px - -.. _appendix_pat_token: - -Appendix 2. Setup ``PAT_TOKEN`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The GitHub workflow needs to link with this privilege through a personal access token (PAT) of the admin (Prof. Billinge). - -1. Visit https://github.com/settings/tokens - -2. Click ``Generate new token`` and choose the classic option. - -3. Under ``Note``, write, "GitHub CI release" - -4. Set the Expiration date of the token. - -5. Under ``Select scopes``, check ``repo`` and ``user``. - -6. Scroll down, click ``Generate token``. - -7. Done! - -.. image:: ./img/add-personal-access-token.png - :alt: add-personal-access-token - :width: 600px - -Copy and paste the ``PAT_TOKEN`` to your GitHub organization: - -1. Visit ``Settings`` in the organization. - -2. Click the ``Actions`` tab under ``Secrets and variables``. - -3. Click ``New organization secret`` and add a new secret and name it as ``PAT_TOKEN``. - -4. Done! diff --git a/conda_forge_guide.html b/conda-forge-guide.html similarity index 54% rename from conda_forge_guide.html rename to conda-forge-guide.html index 5191fa6..94c0935 100644 --- a/conda_forge_guide.html +++ b/conda-forge-guide.html @@ -19,8 +19,8 @@ - - + + @@ -43,21 +43,27 @@ @@ -76,7 +82,7 @@
  • How to release conda-forge package
  • - View page source + View page source
    • @@ -86,62 +92,85 @@

    How to release conda-forge package

    +
    +

    I already have a conda-forge feedstock. I want to release a new package version. How do I do that?

    +

    Please skip to here

    +
    -

    I am new to conda-forge. How do I create a conda package?

    -

    Here, you will learn how to release a conda package distributed through the conda-forge channel in 10-15 minutes. This guide assumes you are familiar with a basic clone, fork, and pull request (PR) workflow on GitHub.

    -
    -

    Step 1. Prepare meta.yaml. See Appendix 1 to learn more about meta.yaml

    -

    To generate a package, we first need to generate a “recipe” for the package. The recipe contains the type of programming language, the package version, the source code, the dependencies, and license, etc. This recipe is stored in a file called meta.yaml.

    -

    Hence, in Step 1, we will generate meta.yaml using the Billinge group’s template. See https://github.com/conda-forge/diffpy.utils-feedstock/blob/main/recipe/meta.yaml as an example of a meta.yaml used in production.

    -

    If you are interested in learning more about each component within meta.yaml, read Appendix 1 located at the end of this document.

    -
      -

    1. Install cookiecutter via pip install cookiecutter and run cookiecutter https://github.com/billingegroup/staged-recipes-cookiecutter

      2. -

    2. Answer the following questions. Default values in parentheses are used if no value is provided.

      -
      -
        -

      1. github_org (diffpy):

        2. -

      2. module_name (diffpy.my_project): diffpy.srreal

        3. -

      3. repo_name (diffpy.srreal):

        4. -

      4. version (1.0.0): 1.3.1

        5. -

      5. Select source 1 - PyPi or 2 - GitHub: 1

        -
          -

        1. Choose 1. Conda package can be built using PyPI’s sdist containing requirements files, src/tests, and pyproject.toml

          2. -
        -
        6. -

      6. project_short_description (Python package for doing science.):

        7. -

      7. project_full_description (This is a Python package for doing science.):

        8. -

      8. license_file (LICENSE.rst):

        9. -

      9. recipe_maintainers (sbillinge,):

        10. -

      10. build_requirements ():

        +

        I am new to conda-forge. How do I create a conda package?

        +

        Here, you will learn how to release a conda package distributed through the conda-forge channel in 10 to 15 minutes. This guide assumes you are familiar with a basic clone, fork, and pull request workflow on GitHub.

        +
        +

        Overview

        +

        The process is divided into three steps:

          -

        1. copy requirements/build.txt from the project repo.

          2. -

        2. Empty for pure Python packages, otherwise compilers will be required.

          3. +

        3. Prepare recipe: You will learn to prepare package information in a file called meta.yaml using our group’s cookiecutting template. The file serves as a recipe for building your conda package. The recipe contains the package version, the source code, the dependencies, the license, etc.

          4. +

        4. Upload therecipe: Once you have the meta.yaml generated, you will create a pull request the staged-recipe repository in the conda-forge repository here

          5. +

        5. Recipe review: One of the community members of conda-forge will review your meta.yaml and provide feedback. Once the recipe is approved, you will have a package available for conda install automatically, and you will have your own designated feedstock repository that contains meta.yaml in https://github.com/conda-forge/<package-name>-feedstock.

        -
        11. -

      11. host_requirements (python >=3.11, setuptools, setuptools-git-versioning >=2.0, pip,):

        -
          -

        1. copy requirements/host.txt

          2. -
        -
        12. -

      12. runtime_requirements (python >=3.11, numpy,):

        +
    +
    +

    1. Prepare conda package recipe in meta.yaml

    +

    To generate a package, we first need to generate a “recipe” for the package. The recipe contains the type of programming language, the package version, the source code, the dependencies, and license, etc. This recipe is stored in a file called meta.yaml.

    +

    Hence, in Step 1, we will generate meta.yaml using the Billinge group’s template. See https://github.com/conda-forge/diffpy.utils-feedstock/blob/main/recipe/meta.yaml as an example of a meta.yaml used in production.

    +

    If you are interested in learning more about each component within meta.yaml, read Appendix 1 located at the end of this document.

      -

    1. copy requirements/conda.txt

      2. -
    - -

  • testing_requirements (pip, pytest,): -1. copy requirements/test.txt

    • +

  • Install cookiecutter via pip install cookiecutter and run cookiecutter https://github.com/billingegroup/staged-recipes-cookiecutter

    • +

  • Answer the following questions. Default values in parentheses are used if no value is provided.

    • +
    +
    +
    github_org:
    +

    The GitHub organization name. For example, diffpy.

    +
    +
    repo_name:
    +

    The name of the repository.

    +
    +
    module_name:
    +

    The name of the module.

    +
    +
    version:
    +

    The version of the package.

    +
    +
    Select:
    +

    Choose PyPI. PyPI’s sdist containing requirements files, src/tests, and pyproject.toml

    +
    +
    short_description:
    +

    A short description of the project

    +
    +
    full_description:
    +

    A full description of the project

    +
    +
    license_file:
    +

    The license file that is located in your project repository. i.g., LICENSE.rst.

    +
    +
    maintainers:
    +

    You may have multiple maintainers sbillinge, bobleesj or just sbillinge

    +
    +
    build_requirements:
    +

    copy requirements/build.txt from the project repo. It should be empty for pure Python packages, otherwise compilers will be required.

    +
    +
    host_requirements:
    +

    Copy the following for the python >=3.11, setuptools, setuptools-git-versioning >=2.0, pip. Copy requirements/host.txt

    +
    +
    runtime_requirements:
    +

    copy requirements/conda.txt:

    +
    +
    testing_requirements:
    +

    copy requirements/test.txt

    +
    +
    - - -

    Now, you have recipes/<package-name>/meta.yaml is generated.

    +

    Now, you have recipes/<package-name>/meta.yaml generated.

    +
    +

    Important

    +
    -
    -

    Step 2. Upload meta.yaml

    +
    +

    2. Upload meta.yaml to conda-forge for initial review

    1. Fork https://github.com/conda-forge/staged-recipes and clone your forked repository

    2. cd into staged-recipes

      3. @@ -155,40 +184,37 @@

      Step 2. Upload

    3. After the CI passes, create a new comment: @conda-forge/help-python Hello Team, ready for review!

    -
    -

    Step 3. Wait for review and merge

    +
    +

    3. Wait for recipe review

    1. Wait for a conda-forge volunteer reviewer to review your submission. It may take up to one week.

    2. Once the PR is merged by the reviewer (1) your package is available on conda-forge, and (2) a new repository will be created under https://github.com/conda-forge/package-name-feedstock/. Example: https://github.com/conda-forge/diffpy.structure-feedstock.

    -
    -

    I already have a conda package. How do I distribute a new package version?

    -

    This step assumes there is a new version of Python package released to PyPI. We will use that PyPI source code to generate a new conda package.

    -

    Obtain the SHA256 value from pypi.org:

    +
    +

    Use the conda-forge feedstock to release a new version

    +

    We release a new package once we have the version and SHA256 sections in meta.yaml in https://github.com/conda-forge/<package-name>-feedstock located in the main branch. The conda-forge team asks to only modify meta.yaml.

    +

    First, we will attain th SHA256 value from pypi.org:

    1. Visit the project on PyPI at https://pypi.org/project/<package-name>

    2. Click Download files under Navigation

    3. Click view hashes under Source Distribution

    4. Copy the SHA256 value

      5. -
    -

    Create a PR to your conda-forge feedstock:

    -
      -

    1. Fork the feedstock repository i.g. https://github.com/conda-forge/diffpy.utils-feedstock.

      2. -

    2. Clone the forked repository.

      3. +

    3. Create a PR to the feedstock repository.

      4. +

    4. If you haven’t, fork and clone the forked feedstock repository.

    5. Run git checkout main && git pull upstream main to sync with the main branch.

    6. Run git checkout -b <version-number> to create a new branch.

      7. -

    7. Open recipe/meta.yaml, modify 1) set version and 2) sha256.

      8. -

    8. Run git add recipe/meta.yaml && git commit -m "Release <version-number>".

      9. +

    9. Open recipe/meta.yaml, modify set version and sha256.

      10. +

    10. Run git add recipe/meta.yaml && git commit -m "release: ready for <version-number>".

    11. Run git push --set-upstream origin <version-number>.

      12. -

    12. Create a PR to main, complete the relevant checklists generated in the PR comment, and modify meta.yaml as needed.

      13. -

    13. Wait for the CI to pass and tag Prof. Billinge for review.

      14. -

    14. Once the PR is merged, verify the latest conda-forge package version from the README badge.

      15. +

    15. Create a PR to main, complete the relevant checklists generated in the PR comment.

      16. +

    16. Wait for the CI to pass and tag Project Owner for review.

      17. +

    17. Once the PR is merged, in 20 to 30 minutes, verify the latest conda-forge package version from the README badge or by visiting https://anaconda.org/conda-forge/<package-name>. i.e.g, https://anaconda.org/conda-forge/diffpy.utils.

    -
    -

    I am familiar with the regular conda release process. How do I do pre-release?

    +
    +

    Appendix 1. How do I do pre-release?

    Generate meta.yaml by following Step 1 and Step 2 under conda-forge: release for the first time above. Here are two differences required for pre-release:

    1. Create recipe/conda_build_config.yaml containing:

      @@ -208,8 +234,12 @@

      I already have a conda package. How do I distribute a new package version?

      For more, read the documentation for pre-release: https://conda-forge.org/docs/maintainer/knowledge_base/#pre-release-builds

    -
    -

    Appendix 1. Background info on meta.yml

    +
    +

    Appendix 2. Add a new admin to the conda-forge feedstock

    +

    Check whether you are an admin listed in the meta.yaml in the feedstock repository. Create an issue with the title/comment: @conda-forge-admin, please add user @username. Please see an example issue here.

    +
    +
    +

    Appendix 3. Background info on meta.yml

    The meta.yaml file contains information about dependencies, the package version, the license, the documentation link, and the maintainer(s) of the package. In meta.yaml, there are 3 important keywords under the requirements section: build, host, and run that are used to specify dependencies.

    +