141 lines
4.6 KiB
Markdown
141 lines
4.6 KiB
Markdown
## Building a new pip package
|
|
|
|
### setup.py updates
|
|
First, update the `_VERSION` field in `setup.py` to a new version number.
|
|
|
|
Next, update the `REQUIRED_PACKAGES` list in the same file to ensure that all
|
|
of our dependencies are listed and that they match the versions of the packages
|
|
referenced in the Bazel `WORKSPACE` file. Also check that the correct version of
|
|
tensorflow is listed.
|
|
|
|
### Building the package
|
|
```
|
|
bazel build //magenta/tools/pip:build_pip_package
|
|
bazel-bin/magenta/tools/pip/build_pip_package /tmp/magenta_pkg
|
|
```
|
|
|
|
Before this next step, make sure your preferred virtualenv or conda environment
|
|
is activated.
|
|
|
|
```
|
|
pip install -U /tmp/magenta_pkg/magenta-N.N.N-py2-none-any.whl
|
|
```
|
|
|
|
Next, test that it worked:
|
|
|
|
```
|
|
$ python
|
|
>>> from magenta.music import midi_io
|
|
>>> midi_io.midi_file_to_sequence_proto('test.mid')
|
|
```
|
|
|
|
You should see the NoteSequence representation of the midi file.
|
|
|
|
### Upload the new version to pypi
|
|
```
|
|
twine upload /tmp/magenta_pkg/magenta-N.N.N-py2-none-any.whl
|
|
```
|
|
|
|
After this step, anyone should be able to `pip install magenta` and get the
|
|
latest version.
|
|
|
|
## Adding to the pip package
|
|
|
|
### Libraries
|
|
|
|
As a convention, libraries that we want to expose externally through the pip
|
|
package should be listed as dependencies in otherwise empty `py_library`
|
|
targets that share the same name as their directory. Those targets should then
|
|
be referenced as dependencies by the target for the directory above them, all
|
|
the way up to the root `//magenta` target. (Targets named the same as their
|
|
package are implicit; `//magenta` is short for `//magenta:magenta`.)
|
|
|
|
This allows us to list `//magenta` as the main `py_library` dependency for
|
|
building the pip package and distributes maintenance of public API dependencies
|
|
to each package.
|
|
|
|
For example, to expose `magenta.music.midi_io`, the `magenta/music/BUILD` file will
|
|
have a section that looks like this:
|
|
|
|
```
|
|
# The Magenta public API.
|
|
py_library(
|
|
name = "lib",
|
|
deps = [
|
|
":midi_io",
|
|
],
|
|
)
|
|
```
|
|
|
|
And the `magenta/BUILD` file will have a section that looks like this:
|
|
|
|
```
|
|
# The Magenta public API.
|
|
py_library(
|
|
name = "magenta",
|
|
visibility = ["//magenta:__subpackages__"],
|
|
deps = [
|
|
"//magenta/music",
|
|
],
|
|
)
|
|
```
|
|
|
|
Because we want `import magenta` to make all modules immediately available,
|
|
we also need to create a custom `__init__.py` in the magenta directory and
|
|
reference it in the `srcs` field for the `//magenta:magenta` target.
|
|
|
|
When you add a new module to be exported, you'll also need to add it to this
|
|
`__init__.py` file. There are instructions in that file for how to
|
|
automatically generate the list based on the python library dependencies.
|
|
|
|
Now the `//magenta/tools/pip:build_pip_package` target just needs to depend on
|
|
`//magenta`.
|
|
|
|
The `//magenta` dependency also provides an easy way to verify that the models
|
|
developed within the magenta repo use the same code that is available to
|
|
external developers. Rather than depend directly on library targets, models
|
|
should depend only on the `//magenta` target.
|
|
|
|
Libraries should continue to use dependencies like normal: one target for every
|
|
python file, and every `import` statement should have a corresponding
|
|
dependency. This ensures we avoid circular dependencies and also makes builds
|
|
faster for tests.
|
|
|
|
### Scripts
|
|
|
|
Our pip package also includes several executable scripts (e.g.,
|
|
`convert_midi_dir_to_note_sequences`). These are just python files that pip
|
|
create executable wrappers around and installs to the python binary path. After
|
|
installation, users will have the script installed in their path. To add a new
|
|
script to the distribution, follow these steps:
|
|
|
|
First, add the script as a data dependency to the
|
|
`//magenta/tools/pip:build_pip_package` target. You will likely also need to
|
|
modify the visibility of the script's target so the pip builder can see it by
|
|
adding this line to the script's target:
|
|
|
|
```
|
|
visibility = ["//magenta/tools/pip:__subpackages__"],
|
|
```
|
|
|
|
Next, modify `setup.py` so the script's python module is listed under
|
|
`CONSOLE_SCRIPTS`.
|
|
|
|
Finally, you will need to modify the script itself so that it can be invoked
|
|
either directly or by the pip-generated wrapper script. The pip wrapper will
|
|
look for a method called `console_entry_point` as defined in `setup.py`, but
|
|
running the script directly (e.g., after a bazel build'ing it) will just invoke
|
|
the `if __name__ == '__main__':` condition. Both of those need to trigger
|
|
`tf.app.run` to run the actual `main` function because `tf.app.run` takes care
|
|
of things like initializing flags. The easiest way to do this is by adding the
|
|
following snippet to the end of your script:
|
|
|
|
```python
|
|
def console_entry_point():
|
|
tf.app.run(main)
|
|
|
|
|
|
if __name__ == '__main__':
|
|
console_entry_point()
|
|
```
|