AI Deploy - Tutorial - Deploy Whisper Speech Recognition Model

Introduction

OpenAI's Whisper large-v3 is the latest iteration of the open-source Whisper Automatic Speech Recognition (ASR) model.

Released on November 6, 2023, Whisper large-v3 has gained recognition in various benchmarks as the top-performing automatic speech recognition (ASR) model. This is why it represents the state of the art in the ASR field. More generally, Whisper has become a powerful tool for transcribing speech in approximately 100 languages, including non-english speech transcription and translation into English.

Objective

In this tutorial, we will guide you through the process of deploying the Whisper large-v3 model in a simple application on AI Deploy for production use.

Unlike a locally hosted application, using AI Deploy offers extremely fast inference since it is deployed on powerful resources (GPUs), and other features like sharing capabilities. You will also learn how to build and use a custom Docker image as part of the deployment process.

Requirements

To build and deploy your Whisper app, you need:

• Access to the OVHcloud Control Panel
• An AI Deploy Project created inside a Public Cloud project in your OVHcloud account
• A user for AI Deploy & Object Storage
• The OVHcloud AI CLI installed on your local computer
• Docker installed on your local computer, or access to a Debian Docker Instance, which is available on the Public Cloud

Instructions

You are going to follow different steps to deploy your Whisper application:

• Whisper Model Selection
• Download Whisper Model in the Object Storage
• Whisper app development
• Whisper app deployment

Step 1: Whisper Model selection

While there are various implementations of the Whisper model (Distil-Whisper, WhisperX, faster-whisper, Whisper JAX, ...), we will use the original Whisper implementation for this tutorial. Feel free to explore other implementations based on your needs (memory constraints and desired inference speed (live transcription or post-transcription)).

Original Whisper implementation comes in five distinct model sizes (tiny, base, small, medium, and large), each designed with specific use cases in mind. Each size has an English-only counterpart (tiny.en for tiny size for example), which demonstrates optimized performance for English applications, outperforming the multilingual variant.

Memory requirements and relative inference speeds are key considerations when selecting your Whisper model. The tiny and base models, requiring approximately 1 GB of VRAM, offer speeds up to 32x and 16x, respectively, compared to the large model. As the models increase in size, the required VRAM and relative speed change accordingly.

In this tutorial, we will deploy the latest version of the large Whisper model in order to obtain the best possible transcription quality, whatever the language spoken. However, you can easily change to another model.

Step 2: Download Whisper Model in the Object Storage

Once you have selected your model, it is time to download it for use in inference.

This can be done within the deployed application, by installing the open-whisper library and executing the following python code that downloads the Whisper model:

pip install -U openai-whisper

import whisper
model_path = "whisper_model"
model_id = 'large-v3'

# Download model 
model = whisper.load_model(model_id, download_root=model_path)

As AI Deploy is based on Docker images, it is advisable to avoid directly downloading the model within the Python code and the Docker image for better manageability and flexibility. Moreover, you should not store the Whisper model directly in the Docker image because it will significantly increase the size of the Docker image.

To do things more efficiently, it is better to save the model in a remote storage, like the OVHcloud Object Storage. This storage will be linked to the AI Deploy app, which will allow the use of the Whisper model within the app. This way, you can easily access the model and make updates without messing with the Docker container itself.

Create an Object Storage bucket for you Whisper model

You can create your Object Storage bucket using either the UI (OVHcloud Control Panel) or the ovhai CLI, which can be downloaded here.

ovhai bucket create <datastore_alias> <bucket_name>

ovhai datastore list

Download whisper in the created bucket

To download the model, we will use AI Training. The created job will be based on an official OVHcloud Docker image ovhcom/ai-training-pytorch which is compliant with the product.

Using 5 CPUs will be enough to download the model, ensuring sufficient memory resources are available.

Two volumes will be associated with this job:

• The first volume will contain the ai-training-examples GitHub repository, containing the Python script used to download the Whisper model. For security, this volume is configured with Read-Only (RO) permission, as we just need access to the Python script, located at ai-training-examples/apps/streamlit/whisper/download_model.py.
• The second volume corresponds to the created bucket, where we will store the Whisper model once downloaded. Therefore, this bucket needs to be with Read-Write (RW) permission, since we will write in it. Make sure to name it correctly, and specify the alias you used during bucket creation.

To launch this job, we will use a pip install command, to install the necessary libraries (from the ai-training-examples/apps/streamlit/whisper/requirements.txt file) and, subsequently, we will execute the python script for Whisper download.

These steps can be summed up in the following command, which contains all the points mentioned above:

ovhai job run ovhcom/ai-training-pytorch \
    --cpu 5 \
    --volume https://github.com/ovh/ai-training-examples.git:/workspace/github_repo:RO \
    --volume whisper-model@GRA/:/workspace/whisper-model:RW \
    -- bash -c 'pip install -r /workspace/github_repo/apps/streamlit/whisper/requirements.txt && python /workspace/github_repo/apps/streamlit/whisper/download_model.py large-v3 /workspace/whisper-model'

The job will then be launched. It will take a few minutes for the two volumes to be added to the job, the environment to be installed and the Whisper model to be downloaded and synchronized with your bucket (FINALZING status).

If you have configured your volumes correctly with the right permissions, and given the right paths to the python script, then you should see your Whisper model in your bucket. This can be checked on the Control Panel, or with the CLI with the following command that will list all the objects of your bucket:

`ovhai bucket object list <bucket_name>@<datastore_alias>`

*Following the example given in this tutorial, we will use:

`ovhai bucket object list whisper-model@GRA`

Step 3: Whisper app development

Now that the Whisper model is in a bucket, you can easily attach it to an AI Deploy app, using the same method as you just did with AI Training.

But in order to interact with the model, we are going to build a very simple application, using Streamlit, which will allow a user to upload an audio file and then transcribe it.

Building a Docker image

As for AI Training, you will need to use a Docker image when using AI Deploy. However you can't use the ovhcom/ai-training-pytorch Docker image this time. Indeed, using the Whisper model requires some system packages to be installed, such as ffmpeg, which is not the case in the previous image. This is why you need to build a new Docker image, dedicated to your AI project.

To do this, you will need Docker, either installed directly on your computer, or using a Debian Docker Instance, available on the Public Cloud.

The Dockerfile to use is already provided. Clone the ai-training-examples GitHub repository:

git clone https://github.com/ovh/ai-training-examples.git

Then run cd ai-training-examples/apps/streamlit/whisper/ to move to the Whisper app folder. If you run ls to list the existing files, you should see several ones, including the following:

• app.py: The python code of the Whisper application.
• requirements.txt: Contains the libraries needed by the Whisper application (streamlit, openai-whisper, ...).
• Dockerfile: Contains all the commands you could call on the command line to run your application (installing requirements.txt, running the python script, ...).

Then, launch one of the following commands to build your application image. The created image will be named whisper_app. You can change it if you wish, but make sure you keep the same identifier throughout the next commands.

# Build the image using your machine's default architecture
docker build . -t whisper_app:latest

# Build image targeting the linux/amd64 architecture
docker buildx build --platform linux/amd64 -t whisper_app:latest .

• The first command builds the image using your system’s default architecture. This may work if your machine already uses the linux/amd64 architecture, which is required to run containers with our AI products. However, on systems with a different architecture (e.g. ARM64 on Apple Silicon), the resulting image will not be compatible and cannot be deployed.

• The second command explicitly targets the linux/AMD64 architecture to ensure compatibility with our AI services. This requires buildx, which is not installed by default. If you haven’t used buildx before, you can install it by running: docker buildx install

During the build process, Docker reads the instructions from the Dockerfile and executes them step by step. For your information, here are the steps of this Dockerfile:

# 🐳 Base image (official small Python image)
FROM python:3.10-slim

# Install missing system packages (ffmpeg is needed for Whisper and is not installed in the small Python image)
RUN apt-get update && \
    apt-get install -y ffmpeg libsndfile1-dev

# 👱 Set the working directory inside the container
WORKDIR /workspace

# 🐍 Copy all folder files (requirements.txt, python code, ...) into the /workspace folder
ADD . /workspace

# 📚 Install the Python dependencies
RUN pip install -r requirements.txt

# 🔑 Give correct access rights to the OVHcloud user
RUN chown -R 42420:42420 /workspace
ENV HOME=/workspace

# 🌐 Set default values for 'model_id' & 'model_path' variables. Will be changeable using --env parameter when launching AI Deploy app
ENV MODEL_ID="large-v3"
ENV MODEL_PATH="/workspace/whisper-model"

# 🚀 Define the command to run the Streamlit application when the container is launched
CMD [ "streamlit" , "run" , "/workspace/app.py", "--server.address=0.0.0.0", "$MODEL_ID", "$MODEL_PATH"]

Push the image into a registry

Once your image is built, you will need to tag it and push it to a registry. Several registries can be used (OVHcloud Managed Private Registry, Docker Hub, GitHub packages, ...). In this tutorial, we will use the OVHcloud shared registry.

You can find the address of your shared registry by launching this command:

ovhai registry list

Log in on your shared registry with your usual AI Platform user credentials:

docker login -u <user> -p <password> <shared-registry-address>

Tag the compiled image and push it into your shared registry:

docker tag whisper_app:latest <shared-registry-address>/whisper_app:latest
docker push <shared-registry-address>/whisper_app:latest

Step 4: Whisper app deployment

Once your image has been pushed, it can be used to deploy new AI solutions.

Run the following command to deploy your Whisper speech-to-text application by running your customised Docker image:

ovhai app run <shared-registry-address>/whisper_app:latest \
      --name whisper_app \
      --gpu 1 \
      --default-http-port 8501 \
      --volume whisper-model@GRA/:/workspace/whisper-model:RO \
      --env MODEL_ID="large-v3" \
      --env MODEL_PATH="/workspace/whisper-model"

Once your application is in RUNNING status, you can transcribe audio files using the powerful Whisper model!

Go further

This tutorial has walked you through deploying the openAI Whisper model implementation for speech-to-text transcription in a Streamlit app.

To continue your project, you can add many more features, such as the specification of the spoken language, or options for translating the transcript and many more!

You could also combine your transcripts with an LLM. Learn how to Deploy LLaMA 2 on AI Deploy.

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Feedback

Please send us your questions, feedback and suggestions to improve the service:

• On the OVHcloud Discord server

*: S3 is a trademark of Amazon Technologies, Inc. OVHcloud’s service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.