PlatformSupportWebsite

Image classifier 101 training job

This document introduces a simple image classifier model, to show how to build the necessary packages to host a model on the Brevetti AI platform. It includes the following necessary steps

  • Model training code that can also produce a user configuration settings_schema.json file

    • should accept cmd args with job_id and api_key or find sagemaker hyperparameter file with the same

    • should produce a model artifact

    • these steps are handled by the brevettiai Job object

  • A simple Dockerfile to run the code - this is how the script is embedded on the platform

Basic keras Image classifier model

As a minimal model example. The code below will serve as a simple model for image classification based on the MobileNet architecture. It accepts any number of classes and the image size may be specified. For training regularization, it includes a Dropout layer. The model and training code can be found on the documentation github repository image_classifier.py

import tensorflow as tf

def build_image_classifier(classes: list, image_shape: tuple):
    # Model backbone is the MobileNetV2
    backbone = tf.keras.applications.MobileNetV2(
        input_shape=image_shape, include_top=False, weights="imagenet"
    )
    # Features are pooled and the output layer consists of a single dense layer
    model = tf.keras.Sequential([
        backbone,
        tf.keras.layers.GlobalMaxPooling2D(),
        tf.keras.layers.Dropout(0.05),
        tf.keras.layers.Dense(len(classes), activation='softmax', name="---".join(classes))
    ])
    # Model is compiled with ```categorical_crossentropy``` loss and reports accuracy metric
    model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.0001),
                  loss='categorical_crossentropy',
                  metrics=['accuracy'])
    return model

Using some default image size and classes a model generated with the above code, and we can verify that it seems to produce a valid keras model.

Model training Job

To train and export the model, construct a training object that uses the following brevettiai components

  • the Job class implements the Job API which gives access to the Brevetti AI platform and the dataset ressources.

  • the Settings object Job.Settings specifies the configurable elements that we will use in the script. The parameters of these objects / variables may be specified by the platform user

  • package_saved_model module saves tensorflow saved_model file as tar gz for the deployment

To install the brevettiai package simply run pip install -U git+https://bitbucket.org/criterionai/core

The training job can be run in a python script; below is shown what is needed to run a training job, and also the TrainingJob class may be used to create the settings_schema.json file needed by the Brevetti AI platform

Model export

NB: this may take 30 s which is deemed too slow for our test pipelines - therefore the code is commented out

Create platform settings file: settings_schema.json

Model Zoo: Host training script on Brevetti AI

The training script needs to be packaged in a way that the platform can create new training jobs, to produce new model artifacts for the platform user. To set up the training script for the model zoo, use the web interface BrevettiAI=>Ressources=>Create model type. This lets you

  • Give the model training script a name

  • Upload the settings_config.json file

  • Link to the docker image with the training code - see below how to create this

Create model Model zoo Screen shot of UI for Create model: platform.brevetti.ai/resources/modeltypes/create and Model Zoo: https://platform.brevetti.ai/models/zoo

Creating the script

When creating a docker image and script file the following needs to be added to the script, to actually execute the code

Using the training script, the settings schema file can then be created by simply calling the script with the --serialize_schema argument.

A basic docker file

This Dockerfile sets up an environment with tensorflow and install the required brevettiai package. Furthermore it sets the entrypoint so it will run using sagemaker called by the Brevetti AI platform.

The build script also produces the settings_schema.json file that is used to create a configurable model for Brevetti AI platform.

Build the docker image and extract the settings_schema.json

If you don't have docker installed, this step can be done on https://labs.play-with-docker.com/

The python image classifier script and the Dockerfile described in the document, can be found on the documentation github repository image_classifier.py and Dockerfile

Download the files and run the following lines

Upload the docker to a container repository

The last step necessary to deploy a docker image is to upload the image to a place that is available to the cloud training job. This can be a public image on Docker hub or a private image hosted by Brevetti AI

It is recommnded to tag the image - the following snippets may be used to push to Dockerhub

Create new "model type"

With the built docker image, and the ´´´settings_schema.json´´´ file the scene is set to create a new model on the Brevetti AI platform BrevettiAI=>Ressources=>Create model type.

Create model training job and Test the docker image anywhere

On the Brevetti AI platform you can choose Model=>create model to configure a new model training job, by selecting training data and setting the parameters that the settings_schema.json specify. The training job is run in the cloud when Start training button is pushed.

The script that is created will runs the model training anywhere, if the model_id and api_key are provided as command arguments.

NB: these flags hwclock -s --privileged are provided to make the image use the proper clock on windows docker.

To explore the code by examples, please run the in the notebook that can be found on colab on this link A Brevetti Ai Image Classifier 101

PreviousTutorialsNextJob API tutorial

Last updated

Was this helpful?