# PyPI Package Boilerplate

This boilerplate defines the basic structure of files needed to package a Python project and publish it for distribution.

## Pypi Cloud Server

Our internal Pypi server is hosted in AWS and persisted to S3. Explore our existing packages and versions at [https://pypi.theorchard.io/\#/](https://pypi.theorchard.io/#/).

## Getting Started

Here is a description of the relevant files or components to copy into your project and/or modify as needed.

* **setup.py** - This is the most important file when setting up a PyPI package.  It

  contains the configuration options needed to create your distribution and

  subsequently publish it.  In this example, it reads the requirements to specify

  install and testing requirements for the package.  In also sets up some basic

  configuration options.  An important option not referenced in this example is

  `entry_points`.  The most popular kind of entry point is the `console_script`

  entry point, which points to a function that you want made available as a

  command-line tool to wherever your package gets installed.

* **setup.cfg** - This is an optional configuration file.  In this example, it's

  used to specify the test path for pytest.  Other options might be to

  add an `easy_install` section with an `index_url` for a specific PyPI served to

  install from.

* **MANIFEST.in** - The `sdist` command puts a minimal default set of files into a

  distribution.  The manifest allows you to include or exclude additional files or

  directories.  In this example, I include all of the txt requirements files and

  exclude the tests directory from distribution. Tests are commonly excluded from

  internal packages because there is no sense in inflating the size of a published

  package which will only be developed on and installed internally.  Developers can

  run tests after cloning the repository which houses the package.

* **\_**_**version\_**_ - The version is specified in the `__init__.py` file of the

  project and referenced by setup.py as the version to be published.

* **.bumpversion.cfg** - This is the configuration file for Bumpversion.

  Python packages can be published to our internal PyPI server with Jenkins.

  To avoid overwriting packages, we are using Bumpversion.  The version will

  be automatically bumped, committed and tagged.

* **.gitignore** - Some exclusions, such as the build, dist and eggs related files

  and directories.

### Run your Setup with Setuptools

Confirm that your package can install its requirements with setuptools.

```text
$ python setup.py install
```

For this example, the `test_suite` is defined in setup.py. The command below runs the test suite \(anything in the tests directory\) through setuptools.

```text
$ python setup.py test
```

In this example, tests are excluded from the published package \(see `MANIFEST.in`\)

The command below will publish the tarball for distribution.

```text
$ python setup.py sdist
```

Running the above `sdist` command will generate two directories in your project:

* **dist** - This directory will contain your tarred package, e.g. `pypi-boilerplate-0.0.1.tar.gz`. This is exactly equivalent to what will get published to the PyPI server by Jenkins.
* **pypi\_boilerplate.egg-info** - This directory contains information about your package. The most relevant file is the `SOURCES.txt` which will tell you all of the files that were included in your package. This is a useful file to help you debug whether your MANIFEST.in is too restrictive or permissive.

```text
(env) C02QL1ENFVH5:pypi mthomas$ ls -la pypi_boilerplate.egg-info/
total 48
drwxr-xr-x   8 mthomas  THEORCHARD\Domain Users  272 Apr  5 15:40 .
drwxr-xr-x  16 mthomas  THEORCHARD\Domain Users  544 Apr  5 15:40 ..
-rw-r--r--   1 mthomas  THEORCHARD\Domain Users  263 Apr  5 15:40 PKG-INFO
-rw-r--r--   1 mthomas  THEORCHARD\Domain Users  335 Apr  5 15:40 SOURCES.txt
-rw-r--r--   1 mthomas  THEORCHARD\Domain Users    1 Apr  5 15:40 dependency_links.txt
-rw-r--r--   1 mthomas  THEORCHARD\Domain Users    1 Apr  5 15:40 not-zip-safe
-rw-r--r--   1 mthomas  THEORCHARD\Domain Users   13 Apr  5 15:40 requires.txt
-rw-r--r--   1 mthomas  THEORCHARD\Domain Users   23 Apr  5 15:40 top_level.txt
```

The dist and egg info files should be excluded from Github \(see .gitignore above\).

## Publish to PyPi

For the purposes of this example, I will use a package called pypi-boilerplate which uses a directory called pypi\_boilerplate.

You can publish any Python package to our internal PyPI server through the [publish-pypi-package Jenkins job](https://pipeline.theorchard.io/job/publish-pypi-package/).

All published packages can be viewed [here](https://pypi.theorchard.io/#/).

You will be able to publish your package by specifying the repository, version \(major, minor, patch\), and the root directory of your package \(i.e. the location of setup.py\). When you first publish your package, a release candidate version \(rc\) will be assigned, the package will be pushed to the Pypi server, the version will be tagged and committed to Github. You will be able to install your release candidate in dependent projects by specifying the Pypi hostname in requirements.txt. See [Dev Flow Diagram](https://drive.google.com/a/theorchard.com/file/d/0Bzfq7KpN6JACVnJ4d3BLQmo4Wk0/view?usp=sharing) for reference.

You can keep publishing new release candidates by building the job and selecting `rc_version`. When you are ready to publish a production version, select `release` in the Jenkins build. See the Bumpversion section for this Readme for more details on versioning.

The package version will get pulled from `pypi_boilerplate/__init__.py` for this project. Bumpversion is used by the Jenkins job to increment versions. The tarball published will be the equivalent of running `sdist` in your local environment.

The Jenkins job has the necessary credentials to connect to our private PyPI server. In order to publish a package to PyPi, the Jenkins job sets up a pypirc file with credentials in its home directory. Here's a sample of what the pypirc looks like when publishing to a pypi server called `pypi-orch`.

```text
[distutils]
index-servers =
  pypi-orch

[pypi-orch]
repository: https://pypi.theorchard.io/pypi/
username: xxxxxxx
password: xxxxxxx
```

The Jenkins job registers our PyPi server, in this case `pypi-orch`.

```text
$ python setup.py register -r pypi-orch
```

Next, the Jenkins job publishes the `pypi-boilerplate` package \(sdist\) to PyPi.

```text
$ python setup.py sdist upload -r pypi-orch
```

### Bumpversion

The Jenkins job is using [Bumpversion](https://pypi.python.org/pypi/bumpversion) to semantically increment versions. First, a release candidate is set which can be iterated on. Then a production version can be published. Here's an example of what's happening behind the scenes:

```text
# sets to 0.0.2-rc0
$ bumpversion patch

# sets to 0.0.2-rc1
$ bumpversion rc_version

# sets to 0.0.2-rc2
$ bumpversion rc_version

# sets to 0.0.2
$ bumpversion release
```

All versions get published to the Pypi server running in AWS Prod.

### Install Package from PyPi

To install the `pypi-boilerplate` package from the `pypi-orch` PyPi server, add the following line to the top of your requirements file.

```text
-i https://pypi.theorchard.io/pypi/
```