Using this template#

Welcome to the developer guidelines! This document is split into two parts:

The repository setup. This section is relevant primarily for the repository maintainer and shows how to connect continuous integration services and documents initial set-up of the repository.
The contributor guide. It contains information relevant to all developers who want to make a contribution.

Setting up the repository#

First commit#

If you are reading this, you should have just completed the repository creation with :

cruft create https://github.com/scverse/cookiecutter-scverse

and you should have

cd cookiecutter-scverse-instance

into the new project directory. Now that you have created a new repository locally, the first step is to push it to github. To do this, you’d have to create a new repository on github. You can follow the instructions directly on github quickstart guide. Since cruft already populated the local repository of your project with all the necessary files, we suggest to NOT initialize the repository with a README.md file or .gitignore, because you might encounter git conflicts on your first push. If you are familiar with git and knows how to handle git conflicts, you can go ahead with your preferred choice.

Note

If you are looking at this document in the cookiecutter-scverse-instance repository documentation, throughout this document the name of the project is cookiecutter-scverse-instance. Otherwise it should be replaced by your new project name: cookiecutter-scverse-instance.

Now that your new project repository has been created on github at https://github.com/scverse/cookiecutter-scverse-instance you can push your first commit to github. To do this, simply follow the instructions on your github repository page or a more verbose walkthrough here:

Assuming you are in /your/path/to/cookiecutter-scverse-instance. Add all files and commit.

# stage all files of your new repo
git add --all
# commit
git commit -m "first commit"

You’ll notice that the command git commit installed a bunch of packages and triggered their execution: those are pre-commit! To read more about what they are and what they do, you can go to the related section Pre-commit checks in this document.

Note

There is a chance that git commit -m "first commit" fails due to the prettier pre-commit formatting the file .cruft.json. No problem, you have just experienced what pre-commit checks do in action. Just go ahead and re-add the modified file and try to commit again:

git add -u # update all tracked file
git commit -m "first commit"

Now that all the files of the newly created project have been committed, go ahead with the remaining steps:

# update the `origin` of your local repo with the remote github link
git remote add origin https://github.com/scverse/cookiecutter-scverse-instance.git
# rename the default branch to main
git branch -M main
# push all your files to remote
git push -u origin main

Your project should be now available at https://github.com/scverse/cookiecutter-scverse-instance. While the repository at this point can be directly used, there are few remaining steps that needs to be done in order to achieve full functionality.

The pyproject.toml file#

Modern Python package management uses a pyproject.toml that was first introduced in PEP 518. This file contains build system requirements and information, which are used by pip to build the package. For more details please have a look at pip’s description of the pyproject.toml file.

Important metadata fields#

The [project] section in the pyproject.toml file defines several important metadata fields that might require editing. For example, the name, description, authors fields could need updates as the project evolves. Especially, the version field needs to be adapted if newer versions of the package are to be released. See Using VCS-based versioning for more details.

Dependency management#

Package dependencies can be added to the dependencies of the [project] section. You can constrain versions using >= and other less useful operators (>, <, <=, ==, !=, and ~=). A common example would be twine>=4.0.2 which requires twine to be installed with at least version 4.0.2 or greater. As another example, if there is a known buggy version, you could exclude it like numpy >=3.0, !=3.0.5.

Further optional dependencies are defined in the [project.optional-dependencies] section such as dependencies only for tests (test). All dependencies listed in such optional dependency groups can then be installed by specifying them like: pip install <package-name>[test].

Tool configurations#

The pyproject.toml file also serves as single configuration file for many tools such as many Pre-commit checks. For example, the line length for auto-formatting can be configured as follows:

[tool.ruff]
line-length = 120

Coverage tests with Codecov#

Coverage tells what fraction of the code is “covered” by unit tests, thereby encouraging contributors to write tests. To enable coverage checks, head over to codecov and sign in with your GitHub account. You’ll find more information in “getting started” section of the codecov docs.

In the Actions tab of your projects’ github repository, you can see that the workflows are failing due to the Upload coverage step. The error message in the workflow should display something like:

...
    Retrying 5/5 in 2s..
    {'detail': ErrorDetail(string='Could not find a repository, try using repo upload token', code='not_found')}
Error: 404 Client Error: Not Found for url:
...

While codecov docs has a very extensive documentation on how to get started, if you are using the default settings of this template we can assume that you are using codecov in a github action workflow and hence you can make use of the codecov bot.

To set it up, simply go to the codecov app page and follow the instructions to activate it for your repository. Once the activation is completed, go back to the Actions tab and re-run the failing workflows.

The workflows should now succeed, and you will be able to find the code coverage at this link: https://app.codecov.io/gh/scverse/cookiecutter-scverse-instance. You might have to wait a couple of minutes and the coverage of this repository should be ~60%.

If your repository is private, you will have to specify an additional token in the repository secrets. In brief, you need to:

Generate a Codecov Token by clicking setup repo in the codecov dashboard.
- If you have already set up codecov in the repository by following the previous steps, you can directly go to the codecov repo webpage.
Go to Settings and copy only the token _______-____-....
Go to Settings of your newly created repository on GitHub.
Go to Security > Secrets > Actions.
Create new repository secret with name CODECOV_TOKEN and paste the token generated by codecov.

Past these additional lines in /.github/workflows.test.yaml under the Upload coverage step:

- name: Upload coverage
  uses: codecov/codecov-action@v3
  with:
      token: ${{ secrets.CODECOV_TOKEN }}    ```

Go back to github Actions page an re-run previously failed jobs.

Documentation on readthedocs#

We recommend using readthedocs.org (RTD) to build and host the documentation for your project. To enable readthedocs, head over to their website and sign in with your GitHub account. On the RTD dashboard choose “Import a Project” and follow the instructions to add your repository.

Make sure to choose the correct name of the default branch. On GitHub, the name of the default branch should be main (it has recently changed from master to main).
We recommend enabling documentation builds for pull requests (PRs). This ensures that a PR doesn’t introduce changes that break the documentation. To do so, got to Admin -> Advanced Settings, check the Build pull requests for this projects option, and click Save. For more information, please refer to the official RTD documentation.
If you find the RTD builds are failing, you can disable the fail_on_warning option in .readthedocs.yaml.

If your project is private, there are ways to enable docs rendering on readthedocs.org but it is more cumbersome and requires a different subscription for read the docs. See a guide here.

Automating PyPI released using GitHub actions#

Configuring the Github workflow#

Tags adhering to "*.*.*" that are pushed to the main branch will trigger the release Github workflow that automatically builds and uploads the Python package to [PyPI][].

For this to work, you’ll need to setup GitHub as a trusted publisher on PyPI. To set this up, login to [PyPI][], and navigate to your project. In the left sidebar, choose “Publishing”, and add the repository details. The “Workflow name” needs to bet set to release.yaml. In most cases, you can leave the “Environment name” empty. For more details, please refer to the official PyPI guide for setting up trusted publishing.

Behind the scenes#

This section explains how releases can be created manually purely for educational purposes. Experienced developers may skip this section. Python packages are not distributed as source code, but as distributions. The most common distribution format is the so-called wheel. To build a wheel, run

python -m build

This command creates a source archive and a wheel, which are required for publishing your package to [PyPI][]. These files are created directly in the root of the repository.

Before uploading them to [PyPI][] you can check that your distribution is valid by running:

twine check dist/*

and finally publishing it with:

twine upload dist/*

Provide your username and password when requested and then go check out your package on [PyPI][]!

For more information, follow the Python packaging tutorial.

Pre-commit checks#

Pre-commit checks are fast programs that check code for errors, inconsistencies and code styles, before the code is committed.

This template uses a number of pre-commit checks. In this section we’ll detail what is used, where they’re defined, and how to modify these checks.

Pre-commit CI#

We recommend setting up pre-commit.ci to enforce consistency checks on every commit and pull-request.

To do so, head over to pre-commit.ci and click “Sign In With GitHub”. Follow the instructions to enable pre-commit.ci for your account or your organization. You may choose to enable the service for an entire organization or on a per-repository basis.

Once authorized, pre-commit.ci should automatically be activated.

Overview of pre-commit hooks used by the template#

The following pre-commit hooks are for code style and format:

prettier: standard code formatter for non-Python files (e.g. YAML).
ruff formatting (ruff-format)
ruff based checks:
- isort (rule category: I): sort module imports into sections and types.
- pydocstyle (rule category: D): pydocstyle extension of flake8.
- flake8-tidy-imports (rule category: TID): tidy module imports.
- flake8-comprehensions (rule category: C4): write better list/set/dict comprehensions.
- pyupgrade (rule category: UP): upgrade syntax for newer versions of the language.

The following pre-commit hooks are for errors and inconsistencies:

pre-commit-hooks: generic pre-commit hooks for text files.
- detect-private-key: checks for the existence of private keys.
- check-ast: check whether files parse as valid python.
- end-of-file-fixer: check files end in a newline and only a newline.
- mixed-line-ending: checks mixed line ending.
- trailing-whitespace: trims trailing whitespace.
- check-case-conflict: check files that would conflict with case-insensitive file systems.
- forbid-to-commit: Make sure that *.rej files cannot be commited. These files are created by the automated template sync if there’s a merge conflict and need to be addressed manually.
ruff based checks:
- pyflakes (rule category: F): various checks for errors.
- pycodestyle (rule category: E, W): various checks for errors.
- flake8-bugbear (rule category: B): find possible bugs and design issues in program.
- flake8-blind-except (rule category: BLE): checks for blind, catch-all except statements.
- Ruff-specific rules (rule category: RUF):
  - RUF100: remove unneccesary # noqa comments ()

How to add or remove pre-commit checks#

The pre-commit checks check for both correctness and stylistic errors. In some cases it might overshoot and you may have good reasons to ignore certain warnings. This section shows you where these checks are defined, and how to enable/ disable them.

pre-commit#

You can add or remove pre-commit checks by simply deleting relevant lines in the .pre-commit-config.yaml file under the repository root. Some pre-commit checks have additional options that can be specified either in the pyproject.toml (for ruff) or tool-specific config files, such as .prettierrc.yml for prettier.

Ruff#

This template configures ruff through the [tool.ruff] entry in the pyproject.toml. For further information ruff configuration, see the docs.

Ruff assigns code to the rules it checks (e.g. E401) and groups them under a rule category (e.g. E). Rule categories are selectively enabled by including them under the select key:

[tool.ruff]
# ...

select = [
    "F",  # Errors detected by Pyflakes
    "E",  # Error detected by Pycodestyle
    "W",  # Warning detected by Pycodestyle
    "I",  # isort
    # ...
]

The ignore entry is used to disable specific rules for the entire project. Add the rule code(s) you want to ignore and don’t forget to add a comment explaining why. You can find a long list of checks that this template disables by default sitting there already.

ignore = [
    # ...
    # __magic__ methods are often self-explanatory, allow missing docstrings
    "D105",
    # ...
]

Checks can be ignored per-file (or glob pattern) with [tool.ruff.per-file-ignores].

[tool.ruff.per-file-ignores]
"docs/*" = ["I"]
"tests/*" = ["D"]
"*/__init__.py" = ["F401"]

To ignore a specific rule on a per-case basis, you can add a # noqa: <rule>[, <rule>, …] comment to the offending line. Specify the rule code(s) to ignore, with e.g. # noqa: E731. Check the Ruff guide for reference.

Note

The RUF100 check will remove rule codes that are no longer necessary from noqa comments. If you want to add a code that comes from a tool other than Ruff, add it to Ruff’s external = [...] setting to prevent RUF100 from removing it.

API design#

Scverse ecosystem packages should operate on AnnData and/or MuData data structures and typically use an API as originally introduced by scanpy with the following submodules:

pp for preprocessing
tl for tools (that, compared to pp generate interpretable output, often associated with a corresponding plotting function)
pl for plotting functions

You may add additional submodules as appropriate. While we encourage to follow a scanpy-like API for ecosystem packages, there may also be good reasons to choose a different approach, e.g. using an object-oriented API.

Using VCS-based versioning#

By default, the template uses hard-coded version numbers that are set in pyproject.toml. If you prefer to have your project automatically infer version numbers from git tags, it is straightforward to switch to vcs-based versioning using hatch-vcs.

In pyproject.toml add the following changes, and you are good to go!

--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,11 +1,11 @@
 [build-system]
 build-backend = "hatchling.build"
-requires = ["hatchling"]
+requires = ["hatchling", "hatch-vcs"]


 [project]
 name = "cookiecutter-scverse-instance"
-version = "0.3.1dev"
+dynamic = ["version"]

@@ -60,6 +60,9 @@
+[tool.hatch.version]
+source = "vcs"
+
 [tool.coverage.run]
 source = ["cookiecutter-scverse-instance"]
 omit = [

Don’t forget to update the Making a release section in this document accordingly, after you are done!

Automated template sync#

Automated template sync is enabled by default. This means that every night, a GitHub action runs cruft to check if a new version of the scverse-cookiecutter template got released. If there are any new changes, a pull request proposing these changes is created automatically. This helps keeping the repository up-to-date with the latest coding standards.

It may happen that a template sync results in a merge conflict. If this is the case a *.ref file with the diff is created. You need to manually address these changes and remove the .rej file when you are done. The pull request can only be merged after all *.rej files have been removed.

Tip

The following hints may be useful to work with the template sync:

GitHub automatically disables scheduled actions if there has been not activity to the repository for 60 days. You can re-enable or manually trigger the sync by navigating to Actions -> Sync Template in your GitHub repository.
If you want to ignore certain files from the template update, you can add them to the [tool.cruft] section in the pyproject.toml file in the root of your repository. More details are described in the cruft documentation.
To disable the sync entirely, simply remove the file .github/workflows/sync.yaml.

Moving forward#

You have reached the end of this document. Congratulations! You have successfully set up your project and are ready to start. For everything else related to documentation, code style, testing and publishing your project to pypi, please refer to the contributing docs.

Using this template

Contents

Using this template#

Setting up the repository#

First commit#

The pyproject.toml file#

Important metadata fields#

Dependency management#

Tool configurations#

Coverage tests with Codecov#

Documentation on readthedocs#

Automating PyPI released using GitHub actions#

Configuring the Github workflow#

Behind the scenes#

Pre-commit checks#

Pre-commit CI#

Overview of pre-commit hooks used by the template#

How to add or remove pre-commit checks#

pre-commit#

Ruff#

API design#

Using VCS-based versioning#

Automated template sync#

Moving forward#