Testing in astrodata

Core Concepts

Astrodata uses a combination of pytest and nox for running tests in isolated, reproducible environments. nox is also used for other automation tasks.

astrodata implements different kinds of tests targeting different aspects of the codebase. These tests are run in isolated environments to ensure that the tests are reproducible and that the code is tested in a clean environment.

These tests include:

  • Unit tests: These tests are used to test individual units of code, such as functions or classes. They are used to ensure that the code behaves as expected when given specific inputs.

  • Integration tests: These tests are used to test how different parts of the codebase interact with each other. They are used to ensure that the code behaves as expected when different parts of the codebase are combined.

  • Script tests: These tests are used to test the scripts in the codebase. They are used to ensure that the scripts behave as expected when run. these primarily test example scripts within the documentation.

  • Build/Release tests: These tests are used to test the build and release process, and the integrity of the package being released. They are used to ensure that the code can be built and released correctly.

There is some cross-pollination between these tests, but they are generally separated to ensure ease of finding specific tests, and organizing tests to make understanding and interpretting test results easier.

Running Tests

Running the tests involves some setup. It’s assumed you have already installed and setup Poetry as per the Developer Installation, or are prepared to work with the astrodata source code in some other way.

If you’d like to run the test in your local environment, you can do so by running the following command from the root of the repository:

pytest

This is discouraged, but can be useful for runnin spcific tests or debugging. To Isolate a test, use a keyword and/or point to a file:

pytest -k "test_foo" tests/test_foo.py

The above command will run all tests that match the keyword “test_foo” in their names in the file tests/test_foo.py.

nox is the preferred way to run tests, as it ensures that the tests are run in clean and isolated environments. Simply run:

nox

This will run a subset of the tests recommended for local development, the unit tests, and their coverage. To see all possible tests, run:

nox -l

This will output information about sessions you can select with nox -s. For more information about using nox, see the Nox CLI Documentation.

Note

nox is also used for other automation tasks, such as:

  • building the documentation (nox -s docs)

  • linting & formatting (nox -s linting)

  • creating development environments (nox -s devshell)

Writing Tests

Writing tests is an important part of contributing to astrodata. Tests help ensure that the code behaves as expected, and that changes to the code don’t break existing functionality.

When writing tests, it’s important to follow the testing best practices outlined in the pytest.

Tests are located in the tests/ directory, and are organized by the type of test they are. For example, unit tests are located in the tests/unit/ directory, integration tests are located in the tests/integration/ directory, and so on.

Unit Tests

Unit tests are used to test individual units of code, such as functions or classes. They are used to ensure that the code behaves as expected when given specific inputs. They are also the most common type of test in astrodata, and likely the most common type of test you will write.

Unit tests are in the tests/unit directory. Testing for specific modules are collected in individual files. For example, a test for a function in the astrodata/utils.py module would be located in tests/unit/test_utils.py.

You can run all unit tests (and nothing else) with nox by running:

nox -s unit

Unit tests require that coverage increases if new code is added. It is highly encouraged to write tests for contributions before writing new code. If you get stuck on how to test your idea, feel free to ask for help in the issue or pull request! Testing is a critical part of the development process, and sometimes it’s trickier than we expect.

Integration Tests

Integration tests are used to test how different parts of the codebase interact with each other in a more hollistic way.

Presently, these test uses DRAGONS to test astrodata. This is a work in progress, and will be updated as the testing framework is updated. DRAGONS just happens to be the most convenient way to run this level of testing at the moment.

Script Tests

Warning

Script tests are not yet fully implemented. This section is a placeholder for future development.

Script tests are used to test the scripts in the codebase. They are used to ensure that the scripts behave as expected when run. These tests are primarily used to test example scripts within the documentation.

To run script tests, run:

nox -s scripts

This will go through a list of scripts in pyproject.toml (see the [tool.nox.scripts] section) and run them. If you add a new script to the project, you will need to add it to this list to ensure it is tested.

The process for adding a script to be tested is as follows:

  1. Add the script to the [tool.nox.scripts] section of pyproject.toml. The key (before the equals sign) should be the path to the .rst file with the example to be run, and the value (after the equals sign) should be any arguments to be passed to rst_extract.

    [tool.nox.scripts]
"path/to/script.rst" = "--some-argument"

  2. Run nox -s scripts -- path/to/script.rst to test the script individually. Once you’re happy with the script, you can run nox -s scripts to test all scripts.

Build/Release Tests

Build/release tests test the package being built and sent to, e.g., PyPI. These tests require a devpi server to be running. This is managed by classes in noxfile.py and nox, and is something you should be aware of (though, hopefully, it will not be an issue).

To run all build/release tests, run:

nox -t build_tests

This will run all unit tests and integrations tests using the fresh astrodata build.

These tests take a while, and are readily handled by our GitHub Actions workflows (see build_tests.yml). If you’re working on the build/release process, you may want to run these tests locally, though.