Workarounds in BentoML 0.12

Dr Selfie -

BentoML is an amazing toolkit to solve pain points in the deployment and serving of machine learning projects.

The following workarounds cover the BentoML 0.12, and with the rapid development, BentoML may solve these issues in the future releases. Therefore, these workarounds may only be useful to BentoML 0.12.1 or previous versions.

1. Does Bentoml support multiple model in one bento?

Yes, but it is not recommended to do so.

  • "For most model serving scenarios, we recommend one model per prediction service, and decouple non-related models into separate services. The only exception is when multiple models are depending on each other, such as the example above.
  • Check here Packaging Model Artifacts

2. Do I need write any code for hosting the API when using Bentoml?

No, you don't need.

  • After adding this API decorator bentoml.api() to your prediction class, Bentoml will take care of all API issues.
  • Check API Function and Adapters

3. Why doesn't BentoML copy my script folder to the packed bento folder?

So far, BentoML will only copy relevant python folders with __init__.py file when packing, so please make sure you have __init__.py in each level of your script folder;

  • If you want Bento to copy non-python file when packing: try following this example:

mkdir(saved_path + '/conf') copy("conf/ar_en_wiki_dict", saved_path + '/conf')

4. Without Yatai service, how can we deploy a bento?

Yes you can:

  • pull code from your git repo, and pull model data from the hosting server
  • run your bento packing script in the terminal to pack a bento;
  • run bentoml list command in terminal to check the bento BENTO_ID;
  • go to ~/bentoml/repository/CLASS_NAME/BENTO_ID to locate the above project; e. use docker build command to compile the docker locally to validate the bento;
  • rsync the entire project to remote server, and compile the docker image;

5. How can I install a required dataset of a lib, e.g. NLTK needs to download additional dataset after pip installation.

You need specify this in the env decorator with a customized setup.sh, and the following is an example

@bentoml.env(
requirements_txt_file='bento_requirements.txt',
setup_sh="""\ 
#!/bin/bash
mkdir /home/bentoml/nltk_data
pip install nltk
python -c "import nltk; nltk.download('punkt', download_dir='/home/bentoml/nltk_data')"
"""
)

Check Init Bash Script

6. Why doesn't BentoML download the dataset into the correct path?

BentoML, by default, uses the sudo access to compile a docker image, so any downloaded dataset will be saved to the sudo folder if you don't specify the path explicitly;

  • If you need to know where the correct path is, after compiling the image, you can use docker exec -it CONTAINER_NAME bash to access the terminal of the container, and install the dataset manually to get the correct path;
  • Getting the correct path, you can put this into the setup.sh in the @env decorator as shown in question 5 above

7. How can we install a pip library from github?

  • We need define this in the setup.sh as mentioned in Question 5.
@bentoml.env(
requirements_txt_file='bento_requirements.txt',
setup_sh="""\ 
#!/bin/bash
git clone https://github.com/pytorch/fairseq && cd fairseq && pip install --editable ./
"""
)