Packaging

Software packaging is the process of organizing and bundling software components, libraries, dependencies, and resources into a distributable format, called a package, that can be easily installed and used by end-users.

Python

The Python packaging ecosystem provides several tools and standards to simplify packaging, distribution, and installation of Python software.

When doing python -m [package-name], then does the -m refer to and look for a main function inside of the package. (So the "m" does not really stand for module here.

sys.path shows the paths which Python is looking in to find packages.

Distutils is the default packaging framework included with Python. It provides a basic set of utilities for building packages, creating distribution archives, and installing packages. Distutils is sufficient for simple projects but has limited functionality compared to more advanced packaging tools.

Pyproject.toml

Add Classifiers

Setuptools is an enhanced version of Distutils and is widely used in the Python community. It extends Distutils by adding support for more complex project layouts, package metadata, and dependency management. Setuptools also introduced the concept of "eggs," which are a bundle format for Python distributions.

Entrypoints

Entry points are a type of metadata that can be exposed by packages on installation.

Example pyproject.toml file

[build-system]
requires = ['setuptools>=60.2.0', 'setuptools-scm', "mypy"]
build-backend = 'setuptools.build_meta'

[project]
name = "hax"
version = "0.0.3"
description = "Personal Automation."
license = { text = "Proprietary" }
keywords = ["automation", "productivity"]
authors = [
        { name = "Kristian Grönberg", email = "[email protected]" },
]

requires-python = ">=3.12.0"
dependencies = [
        "atlassian-python-api",
        "ipdb",
        "httpie",
        "jupyterlab",
        "keyring",
        "langchain",
        "openai",
        "pdbpp",
        "pipdeptree",
        "python-lsp-ruff",
        "python-lsp-server",
        "rich",
        "toml",
        "tomli"
]

[project.scripts]
h = "hax.cli.main:app"

[tool.setuptools]
package-dir = {"" = "src"}
include-package-data = true

[tool.setuptools.packages.find]
where = ["src"]

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
]

[tool.ruff]
exclude = [
    ".bzr",
    ".direnv",
    ".eggs",
    ".git",
        ".ipynb_checkpoints",
    ".mypy_cache",
    ".nox",
    ".pants.d",
    ".pytype",
        ".pytest_cache",
    ".ruff_cache",
    ".svn",
    ".tox",
        ".virtual_documents",
    "__pypackages__",
    "build",
    "dist",
        "ledger",
    "node_modules",
        "org",
        "site"
]

line-length = 120
indent-width = 4
target-version = "py311"

[tool.ruff.lint]
# Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`) codes by default.
# Unlike Flake8, Ruff doesn't enable pycodestyle warnings (`W`) or
# McCabe complexity (`C901`) by default.
select = ["E4", "E7", "E9", "F"]
ignore = []
extend-select = [
  "UP",  # pyupgrade
  "D",   # pydocstyle
]
# Allow fix for all enabled rules (when `--fix`) is provided.
fixable = ["ALL"]
unfixable = []

dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"

[tool.ruff.lint.pydocstyle]
convention = "google"

[tool.ruff.format]
quote-style = "double"
indent-style = "space"
skip-magic-trailing-comma = false
line-ending = "auto"

Pip

pip is a package manager for Python packages, or modules if you like. A package contains all the files you need for a module. Use PyPi to find packages.

The Python Package Index (PyPI) is a central repository for hosting and distributing Python packages. It serves as a trusted source for developers to share their code, making it easily installable by others. PyPI allows users to search, download, and manage dependencies for Python packages using package managers like pip.

Install with pip

pip install <package> will install a package. pip uninstall <package> will uninstall a package. To list all currently install packages do pip list. To list all outdated packages do pip list --outdated.

To update a package do pip install -U <package_name>. To upgrade pip itself do pip install --upgrade pip.

If pip is missing from an environment then can it be installed with the program python -m ensurepip.

Install from git

To install from a git repository do pip install [Package]@git+[URL of the repository]. For example, pip install pip@git+https://github.com/pypa/pip.

To install a specific version do pip install pip@git+https://github.com/pypa/pip@[version commit]

Connection error SSL CERTIFICATE_VERIFY_FAILED

Depending on if you are behind a (corporate) firewall might it not be possible to reach the pypi servers. One of the most probable causes of this issue is that the company's firewall does not trust Python certificates. To fix this you need to add the following configuration file

Create or update the file. The file is located differently depending on your OS:

MacOS: ~/Library/Application\ Support/pip/pip.conf
Unix: $HOME/.config/pip/pip.conf
Windows: %APPDATA%\pip\pip.ini

Add the following information.

[global]
trusted-host = pypi.python.org
               pypi.org
               files.pythonhosted.org

Requirements file

The requirements file is a way to specify which dependencies a project has. Create the requiremnets file by doing python -m pip freeze > requirements.txt.

To later on install all the modules from a requirements file do pip install -r requirements.txt or do pip install --upgrade -r requirements.txt to also upgrade the packages.

You can specify that a particular module should have the exact version by using the double equal sign (==). If you prefer that pip downloads the latest version of a module then use the greater or equal to sign (>=).

To specify that a version should be below a certain number, then use the less or equal to sign (<=).

To specify that a module should not be equal to a certain version use the not equal sign (!=)

It is possible to give several constraints on the same line by putting a comma between the constraints.

certifi>=2019.11.28, <=2022.10.01, !=2021-11-11

Requirements for development environment
You will have some modules which are only required during development. Those modules could be pytest or MkDocs for example. To separate those dependencies which only are required in development, create another requirements file called requirements<sub>dev.txt</sub>.

To have the same modules mention in the dev requirements file as in the standard requiremnts file do include a `-r requirements.txt` line at the top of the dev requirements file.
```
-r requirements.txt
mkdocs>=1.0.0
```
Freeze requirements for production environment

To always be on the latest version of a particular package might be desired in the development environment, but for the production environment do we generally want to freeze the version numbers.

To do so create a requirements<sub>locked.txt</sub> file and set the version numbering to be of the equal equal (==) type.
Local packages

To install from local folders do run pip install <local_folder>. If you want to be able to update the installed package without having to reinstall, then add the option editable. Using editable will only a list of links be created, pip install -e <local_folder>.

Building a package

Wheels are the preferred binary package format in Python, introduced to replace the older "egg" format. A wheel is essentially a ZIP archive containing the project's code along with its compiled C extensions (if any). They provide faster installation compared to source distributions because they don't require compilation during installation.

Use the build package to build a package with the command python -m build. The newly built packages are placed in sub-folder called dist.

Check that the build is good with twine with the command twine check dist/*.

Upload a package to PyPI

Use twine to updload a package with the command twine upload dist/*.

Virtualenv

Virtualenv is a tool for creating isolated Python environments. It allows developers to have multiple environments with different package versions without conflicts. Virtualenv creates a self-contained Python installation, providing an isolated environment for developing and testing Python applications.

PyPI Packaging Metadata

Python packages contain metadata files (e.g., setup.py, setup.cfg) that describe the project, its dependencies, versions, authors, licenses, and more. These metadata files help package managers and build tools understand how to handle and install the package correctly.

Build Tools

Build tools like setuptools, distutils, or newer tools like flit and poetry automate the packaging and distribution process. They generate the required package metadata, compile extensions (if needed), and create distribution archives or wheels.

Swift

Packages in Swift are managed using the Swift Package Manager (SPM), which is a command-line tool integrated into the Swift language. SPM simplifies package creation, versioning, and dependency management. Through SPM, developers can define dependencies, specify version requirements, and automatically fetch and integrate required dependencies into their projects.

Swift packages follow a specific directory structure and include a manifest file, known as Package.swift. This manifest file provides metadata about the package, its dependencies, target platforms, and build configuration. It allows developers to define package dependencies using various methods, such as Git repositories, local directories, or other package managers.