Exercise – deploying a dummy workflow with Vertex AI Pipelines

Before we continue with the hands-on exercise, let’s understand what Vertex AI Pipelines is. Vertex AI Pipelines is a tool for orchestrating ML workflows. Under the hood, it uses an open source tool called Kubeflow Pipeline. Like the relationship between Airflow and Cloud Composer or Hadoop and Dataproc, to understand Vertex AI Pipelines, we need to be familiar with Kubeflow Pipelines.

Kubeflow Pipelines is a platform for building and deploying portable, scalable ML workflows based on Docker containers. Using containers for ML workflows is particularly important compared to data workflows. For example, in data workflows, it’s typical to load the BigQuery, GCS, and pandas libraries for all the steps. Those libraries will be used in the upstream to downstream steps. In ML, the upstream process is data loading; the other step is building models that need specific libraries, such as TensorFlow or scikit...