Managing Dependencies
Qwak supports a variety of Python frameworks to manage model dependencies.
Poetry
Qwak system uses Poetry version 1.5.1.
Poetry Lock Support
Qwak supports poetry.lock
files as long as they're under the same scope as the pyproject.toml
file.
Given the following model structure:
qwak_based_model/
βββ main/
βββββ pyproject.toml
βββββ poetry.lock
βββ tests/
Both files pyproject.toml
and poetry.lock
will be used by Poetry while executing poetry install
cmd.
Poetry Project Starter
Example of quick project starter
[tool.poetry]
name = "Qwak-environment"
version = "0.1.0"
description = "Qwak virtual environment"
authors = ["no-reply@localhost>"]
[tool.poetry.dependencies]
python = "~3.9"
qwak-sdk = "*"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
When specifying dependencies in Poetry, using *
as the version for qwak-sdk
instructs Poetry to install the latest available version of the qwak-sdk package. This approach ensures that your project always utilizes the most recent features and fixes. However, it's important to consider the implications of automatically adopting new versions, as they may introduce breaking changes or compatibility issues. For more controlled dependency management, consider the following alternatives:
qwak-sdk = "^0.5.61"
: This specifies that Poetry should install a version of qwak-sdk that is at least as new as0.5.61
but less than the next major version (1.0.0). It allows for updates that include backwards-compatible features and fixes. This approach balances the benefits of receiving updates with the safety of avoiding major changes that could break your project.qwak-sdk = "0.5.61"
: This pinsqwak-sdk
to a specific version, ensuring that your project will always use version0.5.61
of the SDK. This is the safest option if your project depends on the specific behavior of this version, as it eliminates the risk of unexpected changes due to updates. However, it also means that you will not automatically benefit from new features or fixes introduced in later versions.
.qwakignore file
Occasionally, we may want to exclude a file from the Qwak build but keep it in the repository with the model code. In such cases, we should add the .qwakignore
file to the root directory of our project.
In the file, we define the patterns to match files to exclude from the model build.
For example, suppose we have the following file structure:
.qwakignore
main/
__init__.py
model.py
README.md
tests/
test_model.py
research/
paper_a.pdf
paper_b.pdf
if we want to exclude the entire research
directory and the README.md
file from the build, our .qwakignore
file may contain:
research
README.md
Hidden files
By default, Qwak disregards hidden files. Hidden files are files or directories whose names start with a dot (.) in Unix-like operating systems, or they may have the "Hidden" attribute set in Windows. These files are typically used to store configuration data or hold temporary information.
Suppose you have a directory with files and subdirectories, including a hidden file named
.config_file
. Qwak, following its default behavior, will exclude this file from processing when triggering a remote build.
Incorporating Python Dependencies from .whl
Files
.whl
FilesQwak facilitates the use of Python dependencies packaged as .whl
files through requirements.txt
and conda.yaml
for managing dependencies. It's important to note that Poetry does not support dependencies from .whl
files.
- Preparing Your
.whl
Files:
First, ensure your .whl
file(s) are either uploaded with your model code or fetched from external storage. For instructions on uploading additional dependencies, refer to the Qwak CLI documentation (qwak models build --help
). Below is an example directory structure for your model, where main
is uploaded by default and the dep
directory, containing the pandas dependency in a .whl
file, is included via the --dependency-required-folders dep
option in the Qwak command.
/qwak/model_dir/
.
βββ main # Main directory containing core code
β βββ __init__.py # An empty file that indicates this directory is a Python package
β βββ model.py # Defines the Credit Risk Model
β βββ conda.yaml # Conda environment configurationdata
β
βββ dep # Additional dependency directory added with --dependency-required-folders
β βββ pandas-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
β
βββ tests # Empty directory reserved for future test
β βββ ... # Future tests
|
βββ
- Configuring Dependency Management Files:
Conda: Include the .whl
file in your conda.yaml as follows:
name: test_model
channels:
- defaults
- conda-forge
dependencies:
- python=3.9
- pip:
- "/qwak/model_dir/dep/pandas-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
Requirements.txt: Directly reference the .whl
file path:
"/qwak/model_dir/dep/pandas-2.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
- Using the Dependency in Your Code:
Once the dependency is properly configured, you can import and use it in your Python code as usual:
import pandas as pd
Using OpenCV to build a model
If you add the opencv-python
library to your dependencies and import the cv2
module, you will see the following error Exception: Error in importing module libGL.so.1: cannot open shared object file: No such file or directory
.
To fix the problem, we need to modify the base Docker image and use qwakai/qwak:0.0.13-opencv-cpu-py39
as the base image. If you use the GPU instance, you should set qwakai/opencv-gpu-py39
as the base image.
We can do it in two ways
We can add the --base-image qwakai/qwak:0.0.13-opencv-cpu-py39
parameter to the qwak models build
, or we can use the yaml configuration file. The usage of yaml configuration is described in details in our Build Configurations page.
build_env:
docker:
base_image: qwakai/qwak:0.0.13-opencv-cpu-py39
Updated about 2 months ago