sagemaker/mlops/step-functions-data-science-sdk/step_functions_mlworkflow_scikit_learn_data_processing_and_model_evaluation_with_experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon SageMaker Processing と AWS Step Functions Data Science SDK で機械学習ワークフローを構築する\n",
    "\n",
    "Amazon SageMaker Processing を使うと、データの前/後処理やモデル評価のワークロードを Amazon SageMaker platform 上で簡単に実行することができます。Processingジョブは Amazon Simple Storage Service (Amazon S3) から入力データをダウンロードし、処理結果を Amazon S3 にアップロードします。\n",
    "\n",
    "Step Functions SDK は AWS Step Function と Amazon SageMaker を使って、データサイエンティストが機械学習ワークフローを簡単に作成して実行するためのものです。詳しい情報は以下のドキュメントをご参照ください。\n",
    "\n",
    "* [AWS Step Functions](https://aws.amazon.com/step-functions/)\n",
    "* [AWS Step Functions Developer Guide](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html)\n",
    "* [AWS Step Functions Data Science SDK](https://aws-step-functions-data-science-sdk.readthedocs.io)\n",
    "\n",
    "AWS Step Functions Data Science SDK の SageMaker Processing Step [ProcessingStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/stable/sagemaker.html#stepfunctions.steps.sagemaker.ProcessingStep) によって、AWS Step Functions ワークフローで実装された Sageaker Processing を機械学習エンジニアが直接システムに統合することができます。\n",
    "\n",
    "このノートブックは、SageMaker Processing Job を使ってデータの前処理、モデルの学習、モデルの精度評価の機械学習ワークフローを AWS Step Functions Data Science SDK を使って作成する方法をご紹介します。大まかな流れは以下の通りです。\n",
    "\n",
    "1. AWS Step Functions Data Science SDK の `ProcessingStep` を使ってデータの前処理、特徴量エンジニアリング、学習用とテスト用への分割を行う scikit-learn スクリプトを実行する SageMaker Processing Job を実行\n",
    "1. AWS Step Functions Data Science SDK の `TrainingStep` を使って前処理された学習データを使ったモデルの学習を実行\n",
    "1. AWS Step Functions Data Science SDK の `ProcessingStep` を使って前処理したテスト用データを使った学習済モデルの評価を実行\n",
    "1. AWS Step Functions Data Science SDK の `LambdaStep`を使って最新のモデルと過去のモデルの評価指標の比較を実行\n",
    "\n",
    "\n",
    "このノートブックで使用するデータは [Census-Income KDD Dataset](https://archive.ics.uci.edu/ml/datasets/Census-Income+%28KDD%29) です。このデータセットから特徴量を選択し、データクレンジングを実施し、二値分類モデルの利用できる形にデータを変換し、最後にデータを学習用とテスト用に分割します。このノートブックではロジスティック回帰モデルを使って、国勢調査の回答者の収入が 5万ドル以上か 5万ドル未満かを予測します。このデータセットはクラスごとの不均衡が大きく、ほとんどのデータに 5万ドル以下というラベルが付加されています。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "このノートブックを実行するのに必要なライブラリをインストールします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Import the latest sagemaker, stepfunctions and boto3 SDKs\n",
    "import sys\n",
    "\n",
    "!{sys.executable} -m pip install --upgrade pip\n",
    "!{sys.executable} -m pip install -qU awscli boto3 \"sagemaker>=2.0.0\"\n",
    "!{sys.executable} -m pip install -qU \"stepfunctions==2.1.0\"\n",
    "!{sys.executable} -m pip install sagemaker-experiments\n",
    "!{sys.executable} -m pip show sagemaker stepfunctions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 必要なモジュールのインポート\n",
    "\n",
    "同一アカウントで複数の方が同時にこのノートブックを実行する場合は、以下のセルの一番下の行の `user_name` に各自のお名前を設定してください。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import logging\n",
    "import os\n",
    "import random\n",
    "import time\n",
    "\n",
    "import boto3\n",
    "import stepfunctions\n",
    "from stepfunctions import steps\n",
    "from stepfunctions.inputs import ExecutionInput\n",
    "from stepfunctions.steps import (\n",
    "    Chain,\n",
    "    ChoiceRule,\n",
    "    ModelStep,\n",
    "    ProcessingStep,\n",
    "    TrainingStep,\n",
    "    TransformStep,\n",
    ")\n",
    "from stepfunctions.template import TrainingPipeline\n",
    "from stepfunctions.template.utils import replace_parameters_with_jsonpath\n",
    "from stepfunctions.workflow import Workflow\n",
    "\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.amazon.amazon_estimator import image_uris\n",
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from sagemaker.s3 import S3Uploader\n",
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "from sagemaker.analytics import ExperimentAnalytics\n",
    "from smexperiments.experiment import Experiment\n",
    "from smexperiments.trial import Trial\n",
    "from smexperiments.trial_component import TrialComponent\n",
    "from smexperiments.tracker import Tracker\n",
    "\n",
    "from datetime import datetime\n",
    "from dateutil import tz\n",
    "from time import sleep\n",
    "\n",
    "# SageMaker Session\n",
    "sagemaker_session = sagemaker.Session()\n",
    "region = sagemaker_session.boto_region_name\n",
    "\n",
    "# SageMaker Execution Role\n",
    "# You can use sagemaker.get_execution_role() if running inside sagemaker's notebook instance\n",
    "role = get_execution_role()\n",
    "\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "iam_client = boto3.client('iam', region_name=region)\n",
    "lambda_client = boto3.client('lambda', region_name=region)\n",
    "ecr_client = boto3.client('ecr', region_name=region)\n",
    "JST = tz.gettz('Asia/Tokyo')\n",
    "user_name = 'sample'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "次に、ノートブックから Step Functions を実行するための IAM ロール設定を行います。\n",
    "\n",
    "### ノートブックインスタンスの IAM ロールに権限を追加\n",
    "\n",
    "以下の手順を実行して、ノートブックインスタンスに紐づけられた IAM ロールに、AWS Step Functions のワークフローを作成して実行するための権限と Amazon ECR にイメージを push するための権限を追加してください。\n",
    "\n",
    "\n",
    "1. [Amazon SageMaker console](https://console.aws.amazon.com/sagemaker/) を開く\n",
    "1. **ノートブックインスタンス** を開いて現在使用しているノートブックインスタンスを選択する\n",
    "1. **アクセス許可と暗号化** の部分に表示されている IAM ロールへのリンクをクリックする\n",
    "1. IAM ロールの ARN は後で使用するのでメモ帳などにコピーしておく\n",
    "1. **ポリシーをアタッチします** をクリックして `AWSStepFunctionsFullAccess` を検索する\n",
    "1. `AWSStepFunctionsFullAccess` の横のチェックボックスをオンにする\n",
    "1. 同様の手順で以下のポリシーのチェックボックスをオンにして **ポリシーのアタッチ** をクリックする\n",
    "  - AmazonEC2ContainerRegistryFullAccess\n",
    "  - IAMFullAccess\n",
    "  - AWSLambda_FullAccess\n",
    "\n",
    "もしこのノートブックを SageMaker のノートブックインスタンス以外で実行している場合、その環境で AWS CLI 設定を行ってください。詳細は [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) をご参照ください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "次に Step Functions で使用する実行ロールを作成します。\n",
    "\n",
    "### Step Functions の実行ロールの作成\n",
    "\n",
    "作成した Step Functions ワークフローは、AWS の他のサービスと連携するための IAM ロールを必要とします。以下のセルを実行して、必要な権限を持つ IAM Policy を作成し、それを新たに作成した IAM Role にアタッチします。\n",
    "\n",
    "なお、今回は広めの権限を持つ IAM Policy を作成しますが、ベストプラクティスとしては必要なリソースのアクセス権限と必要なアクションのみを有効にします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "step_functions_policy_name = 'AmazonSageMaker-StepFunctionsWorkflowExecutionPolicy-' + user_name\n",
    "inline_policy ={\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"VisualEditor0\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"events:PutTargets\",\n",
    "                \"events:DescribeRule\",\n",
    "                \"events:PutRule\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTrainingJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTransformJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTuningJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForECSTaskRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForBatchJobsRule\"\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"Sid\": \"VisualEditor1\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": \"iam:PassRole\",\n",
    "            \"Resource\": role,\n",
    "            \"Condition\": {\n",
    "                \"StringEquals\": {\n",
    "                    \"iam:PassedToService\": \"sagemaker.amazonaws.com\"\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"Sid\": \"VisualEditor2\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"batch:DescribeJobs\",\n",
    "                \"batch:SubmitJob\",\n",
    "                \"batch:TerminateJob\",\n",
    "                \"dynamodb:DeleteItem\",\n",
    "                \"dynamodb:GetItem\",\n",
    "                \"dynamodb:PutItem\",\n",
    "                \"dynamodb:UpdateItem\",\n",
    "                \"ecs:DescribeTasks\",\n",
    "                \"ecs:RunTask\",\n",
    "                \"ecs:StopTask\",\n",
    "                \"glue:BatchStopJobRun\",\n",
    "                \"glue:GetJobRun\",\n",
    "                \"glue:GetJobRuns\",\n",
    "                \"glue:StartJobRun\",\n",
    "                \"lambda:InvokeFunction\",\n",
    "                \"sagemaker:AddTags\",\n",
    "                \"sagemaker:CreateEndpoint\",\n",
    "                \"sagemaker:CreateEndpointConfig\",\n",
    "                \"sagemaker:CreateHyperParameterTuningJob\",\n",
    "                \"sagemaker:CreateModel\",\n",
    "                \"sagemaker:CreateProcessingJob\",\n",
    "                \"sagemaker:CreateTrainingJob\",\n",
    "                \"sagemaker:CreateTransformJob\",\n",
    "                \"sagemaker:DeleteEndpoint\",\n",
    "                \"sagemaker:DeleteEndpointConfig\",\n",
    "                \"sagemaker:DescribeHyperParameterTuningJob\",\n",
    "                \"sagemaker:DescribeProcessingJob\",\n",
    "                \"sagemaker:DescribeTrainingJob\",\n",
    "                \"sagemaker:DescribeTransformJob\",\n",
    "                \"sagemaker:ListProcessingJobs\",\n",
    "                \"sagemaker:ListTags\",\n",
    "                \"sagemaker:StopHyperParameterTuningJob\",\n",
    "                \"sagemaker:StopProcessingJob\",\n",
    "                \"sagemaker:StopTrainingJob\",\n",
    "                \"sagemaker:StopTransformJob\",\n",
    "                \"sagemaker:UpdateEndpoint\",\n",
    "                \"sns:Publish\",\n",
    "                \"sqs:SendMessage\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "response = iam_client.create_policy(\n",
    "    PolicyName=step_functions_policy_name,\n",
    "    PolicyDocument=json.dumps(inline_policy),\n",
    ")\n",
    "\n",
    "step_functions_policy_arn = response['Policy']['Arn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "step_functions_role_name = 'AmazonSageMaker-StepFunctionsWorkflowExecutionRole-' + user_name\n",
    "assume_role_policy = {\n",
    "      \"Version\": \"2012-10-17\",\n",
    "      \"Statement\": [{\"Sid\": \"\",\"Effect\": \"Allow\",\"Principal\": {\"Service\":\"states.amazonaws.com\"},\"Action\": \"sts:AssumeRole\"}]\n",
    "    }\n",
    "response = iam_client.create_role(\n",
    "    Path = '/service-role/',\n",
    "    RoleName = step_functions_role_name,\n",
    "    AssumeRolePolicyDocument = json.dumps(assume_role_policy),\n",
    "    MaxSessionDuration=3600*12 # 12 hours\n",
    ")\n",
    "\n",
    "step_functions_role_arn = response['Role']['Arn']\n",
    "\n",
    "response = iam_client.attach_role_policy(\n",
    "    RoleName=step_functions_role_name,\n",
    "    PolicyArn=step_functions_policy_arn\n",
    ")\n",
    "\n",
    "response = iam_client.attach_role_policy(\n",
    "    RoleName=step_functions_role_name,\n",
    "    PolicyArn='arn:aws:iam::aws:policy/CloudWatchEventsFullAccess'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow_execution_role = step_functions_role_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step Functions ワークフロー実行時の入力スキーマ作成\n",
    "\n",
    "Step Functions ワークフローを実行する際に、パラメタなどを引数として渡すことができます。ここではそれらの引数のスキーマを作成します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SageMaker expects unique names for each job, model and endpoint.\n",
    "# If these names are not unique the execution will fail. Pass these\n",
    "# dynamically for each execution using placeholders.\n",
    "execution_input = ExecutionInput(\n",
    "    schema={\n",
    "        \"PreprocessingJobName\": str,\n",
    "        \"PreprocessingInputData\": str,\n",
    "        \"PreprocessingOutputDataTrain\": str,\n",
    "        \"PreprocessingOutputDataTest\": str,\n",
    "        \"TrainingJobName\": str,\n",
    "        \"TrainingParameters\": dict,\n",
    "        \"TrainingOutputModel\": str,\n",
    "        \"ExperimentName\": str,\n",
    "        \"EvaluationScriptPath\": str,\n",
    "        \"EvaluationProcessingJobName\": str,\n",
    "        \"EvaluationProcessingOutput\": str,\n",
    "        \"EvaluationExperimentArgs\": list,\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Amazon SageMaker Experiments のセットアップ\n",
    "\n",
    "このノートブックでは、モデルの評価メトリクスを記録するために Amazon SageMaker Experiments を使用します。`experiment_name` に任意の Experiment 名をセットして以下のセルを実行してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_name = \"sfn-evaluate-model-\" + user_name\n",
    "\n",
    "# create the experiment if it doesn't exist\n",
    "try:\n",
    "    experiment_evaluate = Experiment.load(experiment_name=experiment_name)\n",
    "except Exception as ex:\n",
    "    if \"ResourceNotFound\" in str(ex):\n",
    "        experiment_evaluate = Experiment.create(\n",
    "            experiment_name=experiment_name, \n",
    "            description=\"model evaluation\", \n",
    "            sagemaker_boto_client=boto3.client('sagemaker'))\n",
    "\n",
    "print(experiment_evaluate.experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## データの前処理と特徴量エンジニアリング"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "データクレンジング 、前処理、特徴量エンジニアリングのスクリプトの前に、データセットの初めの 20行をのぞいてみましょう。ターゲット変数は `income` 列です。選択する特徴量は `age`, `education`, `major industry code`, `class of worker`, `num persons worked for employer`, `capital gains`, `capital losses`,  `dividends from stocks` です。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "input_data = \"s3://sagemaker-sample-data-{}/processing/census/census-income.csv\".format(region)\n",
    "df = pd.read_csv(input_data, nrows=10)\n",
    "df.head(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "scikit-learn の前処理スクリプトを実行するために `SKLearnProcessor`を作成します。これは、SageMaker が用意している scikit-learn のコンテナイメージを使って Processing ジョブを実行するためのクラスです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sklearn_processor = SKLearnProcessor(\n",
    "    framework_version=\"0.23-1\",\n",
    "    role=role,\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    instance_count=1,\n",
    "    max_runtime_in_seconds=1200,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下のセルを実行すると `preprocessing.py` が作成されます。これは前処理のためのスクリプトです。以下のセルを書き換えて実行すれば、`preprocessing.py` が上書き保存されます。このスクリプトでは、以下の処理が実行されます。\n",
    "\n",
    "* 重複データやコンフリクトしているデータの削除\n",
    "* ターゲット変数 `income` 列をカテゴリ変数から 2つのラベルを持つ列に変換\n",
    "* `age` と `num persons worked for employer` をビニングして数値からカテゴリ変数に変換\n",
    "* 連続値である`capital gains`, `capital losses`, `dividends from stocks` を学習しやすいようスケーリング\n",
    "* `education`, `major industry code`, `class of worker`を学習しやすいようエンコード\n",
    "* データを学習用とテスト用に分割し特徴量とラベルの値をそれぞれ保存\n",
    "\n",
    "学習スクリプトでは、前処理済みの学習用データとラベル情報を使用してモデルを学習します。また、モデル評価スクリプトでは学習済みモデルと前処理済みのテスト用データトラベル情報を使用してモデルを評価します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile preprocessing.py\n",
    "\n",
    "import argparse\n",
    "import os\n",
    "import warnings\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelBinarizer, KBinsDiscretizer\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "from sklearn.exceptions import DataConversionWarning\n",
    "\n",
    "warnings.filterwarnings(action=\"ignore\", category=DataConversionWarning)\n",
    "\n",
    "\n",
    "columns = [\n",
    "    \"age\",\n",
    "    \"education\",\n",
    "    \"major industry code\",\n",
    "    \"class of worker\",\n",
    "    \"num persons worked for employer\",\n",
    "    \"capital gains\",\n",
    "    \"capital losses\",\n",
    "    \"dividends from stocks\",\n",
    "    \"income\",\n",
    "]\n",
    "class_labels = [\" - 50000.\", \" 50000+.\"]\n",
    "\n",
    "\n",
    "def print_shape(df):\n",
    "    negative_examples, positive_examples = np.bincount(df[\"income\"])\n",
    "    print(\n",
    "        \"Data shape: {}, {} positive examples, {} negative examples\".format(\n",
    "            df.shape, positive_examples, negative_examples\n",
    "        )\n",
    "    )\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"--train-test-split-ratio\", type=float, default=0.3)\n",
    "    args, _ = parser.parse_known_args()\n",
    "\n",
    "    print(\"Received arguments {}\".format(args))\n",
    "\n",
    "    input_data_path = os.path.join(\"/opt/ml/processing/input\", \"census-income.csv\")\n",
    "\n",
    "    print(\"Reading input data from {}\".format(input_data_path))\n",
    "    df = pd.read_csv(input_data_path)\n",
    "    df = pd.DataFrame(data=df, columns=columns)\n",
    "    df.dropna(inplace=True)\n",
    "    df.drop_duplicates(inplace=True)\n",
    "    df.replace(class_labels, [0, 1], inplace=True)\n",
    "\n",
    "    negative_examples, positive_examples = np.bincount(df[\"income\"])\n",
    "    print(\n",
    "        \"Data after cleaning: {}, {} positive examples, {} negative examples\".format(\n",
    "            df.shape, positive_examples, negative_examples\n",
    "        )\n",
    "    )\n",
    "\n",
    "    split_ratio = args.train_test_split_ratio\n",
    "    print(\"Splitting data into train and test sets with ratio {}\".format(split_ratio))\n",
    "    X_train, X_test, y_train, y_test = train_test_split(\n",
    "        df.drop(\"income\", axis=1), df[\"income\"], test_size=split_ratio, random_state=0\n",
    "    )\n",
    "\n",
    "    preprocess = make_column_transformer(\n",
    "        (\n",
    "            KBinsDiscretizer(encode=\"onehot-dense\", n_bins=10),\n",
    "            [\"age\", \"num persons worked for employer\"],\n",
    "        ),\n",
    "        (\n",
    "            StandardScaler(),\n",
    "            [\"capital gains\", \"capital losses\", \"dividends from stocks\"],\n",
    "        ),\n",
    "        (\n",
    "            OneHotEncoder(sparse=False),\n",
    "            [\"education\", \"major industry code\", \"class of worker\"],\n",
    "        ),\n",
    "    )\n",
    "    print(\"Running preprocessing and feature engineering transformations\")\n",
    "    train_features = preprocess.fit_transform(X_train)\n",
    "    test_features = preprocess.transform(X_test)\n",
    "\n",
    "    print(\"Train data shape after preprocessing: {}\".format(train_features.shape))\n",
    "    print(\"Test data shape after preprocessing: {}\".format(test_features.shape))\n",
    "\n",
    "    train_features_output_path = os.path.join(\"/opt/ml/processing/train\", \"train_features.csv\")\n",
    "    train_labels_output_path = os.path.join(\"/opt/ml/processing/train\", \"train_labels.csv\")\n",
    "\n",
    "    test_features_output_path = os.path.join(\"/opt/ml/processing/test\", \"test_features.csv\")\n",
    "    test_labels_output_path = os.path.join(\"/opt/ml/processing/test\", \"test_labels.csv\")\n",
    "\n",
    "    print(\"Saving training features to {}\".format(train_features_output_path))\n",
    "    pd.DataFrame(train_features).to_csv(train_features_output_path, header=False, index=False)\n",
    "\n",
    "    print(\"Saving test features to {}\".format(test_features_output_path))\n",
    "    pd.DataFrame(test_features).to_csv(test_features_output_path, header=False, index=False)\n",
    "\n",
    "    print(\"Saving training labels to {}\".format(train_labels_output_path))\n",
    "    y_train.to_csv(train_labels_output_path, header=False, index=False)\n",
    "\n",
    "    print(\"Saving test labels to {}\".format(test_labels_output_path))\n",
    "    y_test.to_csv(test_labels_output_path, header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前処理用スクリプトを S3 にアップロードします。Processing ジョブの Input にアップロードした S3 パスを指定することで、ジョブ起動後にそのパスから前処理用スクリプトが Processing 用コンテナにダウンロードされます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREPROCESSING_SCRIPT_LOCATION = \"preprocessing.py\"\n",
    "\n",
    "input_code = sagemaker_session.upload_data(\n",
    "    PREPROCESSING_SCRIPT_LOCATION,\n",
    "    bucket=bucket,\n",
    "    key_prefix=user_name + \"/data/sklearn_processing/code\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `ProcessingStep` の作成\n",
    "\n",
    "それでは、SageMaker Processing ジョブを起動するための [ProcessingStep](https://aws-step-functions-data-science-sdk.readthedocs.io/en/stable/sagemaker.html#stepfunctions.steps.sagemaker.ProcessingStep) を作成しましょう。\n",
    "\n",
    "このステップは、前の手順で定義した SKLearnProcessor に入力と出力の情報を追加して使用します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [ProcessingInputs](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.processing.ProcessingInput) と [ProcessingOutputs](https://sagemaker.readthedocs.io/en/stable/api/training/processing.html#sagemaker.processing.ProcessingOutput)  オブジェクトを作成して SageMaker Processing ジョブに入力と出力の情報を追加"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = [\n",
    "    ProcessingInput(\n",
    "        source=execution_input[\"PreprocessingInputData\"], destination=\"/opt/ml/processing/input\", input_name=\"input-1\"\n",
    "    ),\n",
    "    ProcessingInput(\n",
    "        source=input_code,\n",
    "        destination=\"/opt/ml/processing/input/code\",\n",
    "        input_name=\"code\",\n",
    "    ),\n",
    "]\n",
    "\n",
    "outputs = [\n",
    "    ProcessingOutput(\n",
    "        source=\"/opt/ml/processing/train\",\n",
    "        destination=execution_input[\"PreprocessingOutputDataTrain\"],\n",
    "        output_name=\"train_data\",\n",
    "    ),\n",
    "    ProcessingOutput(\n",
    "        source=\"/opt/ml/processing/test\",\n",
    "        destination=execution_input[\"PreprocessingOutputDataTest\"],\n",
    "        output_name=\"test_data\",\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  `ProcessingStep` の作成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_step = ProcessingStep(\n",
    "    \"SageMaker pre-processing step\",\n",
    "    processor=sklearn_processor,\n",
    "    job_name=execution_input[\"PreprocessingJobName\"],\n",
    "    inputs=inputs,\n",
    "    outputs=outputs,\n",
    "    container_arguments=[\"--train-test-split-ratio\", \"0.2\"],\n",
    "    container_entrypoint=[\"python3\", \"/opt/ml/processing/input/code/preprocessing.py\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 前処理済みデータを使ったモデルの学習\n",
    "\n",
    "学習スクリプト `train.py` を使って学習ジョブを実行するための `SKLearn` インスタンスを作成します。これはあとで `TrainingStep` を作成する際に使用します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.sklearn.estimator import SKLearn\n",
    "\n",
    "sklearn = SKLearn(\n",
    "    entry_point=\"train.py\",\n",
    "    train_instance_type=\"ml.m5.xlarge\",\n",
    "    role=role,\n",
    "    framework_version=\"0.23-1\",\n",
    "    py_version=\"py3\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学習スクリプト `train.py` は、ロジスティック回帰モデルを学習し、学習済みモデルを `/opt/ml/model` に保存します。Amazon SageMaker は、学習ジョブの最後に `/opt/ml/model` に保存されているモデルを `model.tar.gz` に圧縮して S3 にアップロードします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile train.py\n",
    "\n",
    "import os\n",
    "import argparse\n",
    "\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import joblib\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    \n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"-c\", type=str, default='1.0')\n",
    "    args, _ = parser.parse_known_args()\n",
    "    \n",
    "    print(args.c)\n",
    "    \n",
    "    training_data_directory = \"/opt/ml/input/data/train\"\n",
    "    train_features_data = os.path.join(training_data_directory, \"train_features.csv\")\n",
    "    train_labels_data = os.path.join(training_data_directory, \"train_labels.csv\")\n",
    "    print(\"Reading input data\")\n",
    "    X_train = pd.read_csv(train_features_data, header=None)\n",
    "    y_train = pd.read_csv(train_labels_data, header=None)\n",
    "\n",
    "    model = LogisticRegression(class_weight=\"balanced\", solver=\"lbfgs\", C=float(args.c), verbose=1)\n",
    "    print(\"Training LR model\")\n",
    "    model.fit(X_train, y_train)\n",
    "    model_output_directory = os.path.join(\"/opt/ml/model\", \"model.joblib\")\n",
    "    print(\"Saving model to {}\".format(model_output_directory))\n",
    "    joblib.dump(model, model_output_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学習用スクリプトを source.tar.gz に固めて S3 にアップロードします。既存のファイルを上書きしてしまわないよう、タイムスタンプを使ってアップロード先のパスを指定します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp = datetime.now(tz=JST).strftime('%Y%m%d%H%M')\n",
    "\n",
    "TRAINNING_SCRIPT_LOCATION = \"source.tar.gz\"\n",
    "!tar zcvf $TRAINNING_SCRIPT_LOCATION train.py\n",
    "\n",
    "train_code = sagemaker_session.upload_data(\n",
    "    TRAINNING_SCRIPT_LOCATION,\n",
    "    bucket=bucket,\n",
    "    key_prefix=os.path.join(user_name, \"data/sklearn_train/code\", timestamp),\n",
    ")\n",
    "train_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `TrainingStep` の作成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_step = steps.TrainingStep(\n",
    "    \"SageMaker Training Step\",\n",
    "    estimator=sklearn,\n",
    "    data={\"train\": sagemaker.TrainingInput(execution_input[\"PreprocessingOutputDataTrain\"], content_type=\"text/csv\")},\n",
    "    job_name=execution_input[\"TrainingJobName\"],\n",
    "    hyperparameters=execution_input[\"TrainingParameters\"],\n",
    "    wait_for_completion=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## モデルの評価\n",
    "\n",
    "`evaluation.py` はモデル評価用のスクリプトです。このスクリプトは scikit-learn と Amazon SageMake Experiments を用いるため、カスタムコンテナを利用できる`ScriptProcessor` を使用します。このスクリプトは学習済みモデルとテスト用データセットを入力として受け取り、各分類クラスの分類評価メトリクス、precision、リコール、F1スコア、accuracy と ROC AUC が記載された JSON ファイルを出力します。\n",
    "\n",
    "パラメタや学習データを変えて複数のモデルを学習させ、それらの精度を比較する際に評価メトリクスの情報を Amazon SageMaker Experiments で管理しておくと、作成した複数のモデルを比較しやすくなります。以下のモデル評価用のスクリプトでは、モデルの評価メトリクスを算出してそれらを Experiment に登録し、また、最新の学習済みモデルが既存のモデルと比べて良いかどうかを評価メトリクスを使って判定し、その結果も Experiment に記録しています。学習ジョブや Processing ジョブの中で Experiments の Tracker を使う際は、`Tracker.load() ` を使用することでジョブが使用している Trial を自動的に読み出して tracking を開始します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile evaluation.py\n",
    "\n",
    "import json\n",
    "import os\n",
    "import tarfile\n",
    "import argparse\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import joblib\n",
    "from sklearn.metrics import classification_report, roc_auc_score, accuracy_score\n",
    "\n",
    "from smexperiments.tracker import Tracker\n",
    "from smexperiments.trial import Trial\n",
    "from sagemaker.analytics import ExperimentAnalytics\n",
    "import sagemaker\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    parser = argparse.ArgumentParser()\n",
    "    \n",
    "    parser.add_argument('--experiment-name', type=str, default=None,\n",
    "                        help='Experiment name')\n",
    "\n",
    "    args = parser.parse_args()\n",
    "    \n",
    "    model_path = os.path.join(\"/opt/ml/processing/model\", \"model.tar.gz\")\n",
    "    print(\"Extracting model from path: {}\".format(model_path))\n",
    "    with tarfile.open(model_path) as tar:\n",
    "        tar.extractall(path=\".\")\n",
    "    print(\"Loading model\")\n",
    "    model = joblib.load(\"model.joblib\")\n",
    "\n",
    "    print(\"Loading test input data\")\n",
    "    test_features_data = os.path.join(\"/opt/ml/processing/test\", \"test_features.csv\")\n",
    "    test_labels_data = os.path.join(\"/opt/ml/processing/test\", \"test_labels.csv\")\n",
    "\n",
    "    X_test = pd.read_csv(test_features_data, header=None)\n",
    "    y_test = pd.read_csv(test_labels_data, header=None)\n",
    "    predictions = model.predict(X_test)\n",
    "\n",
    "    print(\"Creating classification evaluation report\")\n",
    "    report_dict = classification_report(y_test, predictions, output_dict=True)\n",
    "    report_dict[\"accuracy\"] = accuracy_score(y_test, predictions)\n",
    "    report_dict[\"roc_auc\"] = roc_auc_score(y_test, predictions)\n",
    "\n",
    "    print(args.experiment_name)\n",
    "    trial_component_analytics = ExperimentAnalytics(\n",
    "        experiment_name=args.experiment_name,\n",
    "        sort_by=\"parameters.accuracy\",\n",
    "        sort_order=\"Descending\",# Ascending or Descending\n",
    "    )\n",
    "    \n",
    "    df = trial_component_analytics.dataframe()\n",
    "    is_best = 0\n",
    "    try:\n",
    "        best_acc = df.iloc[0]['accuracy'] \n",
    "        if best_acc < report_dict[\"accuracy\"]:\n",
    "            print('This model is the best ever!!')\n",
    "            is_best = 1\n",
    "        else:\n",
    "            print('This model is not so good.')\n",
    "    except:\n",
    "        is_best = 1\n",
    "        print('This model is the first one.')\n",
    "    \n",
    "    print('Recording metrics to Experiments...')\n",
    "    with Tracker.load() as processing_tracker: # Tracker requires with keyword\n",
    "        processing_tracker.log_parameters({ \"accuracy\": report_dict[\"accuracy\"], \n",
    "                                                                           \"roc_auc\": report_dict[\"roc_auc\"], \n",
    "                                                                           \"is_best\": is_best})\n",
    "\n",
    "    print(\"Classification report:\\n{}\".format(report_dict))\n",
    "\n",
    "    evaluation_output_path = os.path.join(\"/opt/ml/processing/evaluation\", \"evaluation.json\")\n",
    "    print(\"Saving classification report to {}\".format(evaluation_output_path))\n",
    "\n",
    "    with open(evaluation_output_path, \"w\") as f:\n",
    "        f.write(json.dumps(report_dict))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "評価用スクリプトを S3 にアップロードします。既存のファイルを上書きしてしまわないよう、タイムスタンプを使ってアップロード先のパスを指定します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp = datetime.now(tz=JST).strftime('%Y%m%d%H%M')\n",
    "\n",
    "MODELEVALUATION_SCRIPT_LOCATION = \"evaluation.py\"\n",
    "\n",
    "input_evaluation_code = sagemaker_session.upload_data(\n",
    "    MODELEVALUATION_SCRIPT_LOCATION,\n",
    "    bucket=bucket,\n",
    "    key_prefix=os.path.join(user_name, \"data/sklearn_processing/code\", timestamp),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "モデル評価用の ProcessingStep の入力と出力オブジェクトを作成します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "inputs_evaluation = [\n",
    "    ProcessingInput(\n",
    "        source=execution_input[\"PreprocessingOutputDataTest\"],\n",
    "        destination=\"/opt/ml/processing/test\",\n",
    "        input_name=\"input-1\",\n",
    "    ),\n",
    "    ProcessingInput(\n",
    "        source=execution_input[\"TrainingOutputModel\"],\n",
    "        destination=\"/opt/ml/processing/model\",\n",
    "        input_name=\"input-2\",\n",
    "    ),\n",
    "    ProcessingInput(\n",
    "        source=execution_input['EvaluationScriptPath'],\n",
    "        destination=\"/opt/ml/processing/input/code\",\n",
    "        input_name=\"code\",\n",
    "    ),\n",
    "]\n",
    "\n",
    "outputs_evaluation = [\n",
    "    ProcessingOutput(\n",
    "        source=\"/opt/ml/processing/evaluation\",\n",
    "        destination=execution_input[\"EvaluationProcessingOutput\"],\n",
    "        output_name=\"evaluation\",\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "モデルの評価結果を SageMaker Experiments で管理するため、Experiments のライブラリを含んだコンテナイメージを作成します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p docker/processing/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dockerfile_name = 'docker/processing/Dockerfile'\n",
    "\n",
    "dockerfile_code = f\"\"\"FROM python:3.7-slim-buster\n",
    "    \n",
    "ENV AWS_DEFAULT_REGION {region}\n",
    "\n",
    "RUN pip3 install --upgrade pip\n",
    "RUN pip3 install -qU boto3 pandas==0.25.3 scikit-learn==0.23.1 sagemaker-experiments==0.1.35 sagemaker==2.109.0\n",
    "\n",
    "ENTRYPOINT [\"python3\", \"/opt/ml/processing/input/code/evaluation.py\"]\n",
    "\"\"\"\n",
    "with open(dockerfile_name, 'w') as f:\n",
    "    f.write(dockerfile_code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ecr_repository = 'sf-model-evaluate-' + user_name\n",
    "tag = ':latest'\n",
    "uri_suffix = 'amazonaws.com'\n",
    "processing_repository_uri = '{}.dkr.ecr.{}.{}/{}'.format(account_id, region, uri_suffix, ecr_repository + tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create ECR repository and push docker image\n",
    "!docker build -t $ecr_repository docker/processing\n",
    "!$(aws ecr get-login --region $region --registry-ids $account_id --no-include-email)\n",
    "!aws ecr create-repository --repository-name $ecr_repository\n",
    "!docker tag {ecr_repository + tag} $processing_repository_uri\n",
    "!docker push $processing_repository_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "データの前処理の際は SKProcessor を使用しましたが、モデルの評価にはカスタムコンテナを使用するため ScriptProcessor を使用します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.processing import ScriptProcessor\n",
    "\n",
    "model_evaluation_processor = ScriptProcessor(\n",
    "    command=['python3'],\n",
    "    image_uri=processing_repository_uri,\n",
    "    role=role,\n",
    "    sagemaker_session=sagemaker_session,\n",
    "    instance_count=1,\n",
    "    instance_type='ml.m5.xlarge',\n",
    "    max_runtime_in_seconds=1200\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `ProcessingStep` の作成\n",
    "\n",
    "ProcessingStep の引数 experiment_config に Experiment 名をセットすることで、起動した Processing ジョブの中で自動的に Trial が作成されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_evaluation_step = ProcessingStep(\n",
    "    \"SageMaker Processing Model Evaluation step\",\n",
    "    processor=model_evaluation_processor,\n",
    "    job_name=execution_input[\"EvaluationProcessingJobName\"],\n",
    "    inputs=inputs_evaluation,\n",
    "    outputs=outputs_evaluation,\n",
    "    experiment_config={\n",
    "             \"ExperimentName\":execution_input[\"ExperimentName\"]},\n",
    "    container_arguments=execution_input[\"EvaluationExperimentArgs\"],\n",
    "    container_entrypoint=[\"python3\", \"/opt/ml/processing/input/code/evaluation.py\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## モデルの評価結果に応じた処理\n",
    "\n",
    "モデルの評価ジョブでの評価結果をもとに、Lambda 関数で後処理を行います。後処理としては例えば、Slack にモデルの評価結果をポストしたり、別の Workflow を起動したりなどが考えられます。このノートブックでは、評価結果に応じて表示するテキストの内容を変えています。\n",
    "\n",
    "このノートブックでは、Experiment で管理されている情報を取得するために `sagemaker.analytics.ExperimentAnalytics` を使用するため、必要なライブラリが入ったコンテナを使って Lambda 関数を実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p docker/lambda/app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ./docker/lambda/app/app.py\n",
    "\n",
    "import sys\n",
    "import time\n",
    "import os\n",
    "import glob\n",
    "import numpy as np\n",
    "import json\n",
    "import boto3\n",
    "from sagemaker.analytics import ExperimentAnalytics\n",
    "\n",
    "def handler(event, context):\n",
    "\n",
    "    experiment_name = event['experiment-name']\n",
    "    job_name = event['evaluation-job-name']\n",
    "    \n",
    "    print('job_name: ', job_name)\n",
    "    \n",
    "    search_expression = {\n",
    "        \"Filters\":[\n",
    "            {\n",
    "                \"Name\": \"TrialComponentName\",\n",
    "                \"Operator\": \"Contains\",\n",
    "                \"Value\": job_name,\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "\n",
    "    trial_component_analytics = ExperimentAnalytics(\n",
    "        experiment_name=experiment_name,\n",
    "        search_expression=search_expression,\n",
    "    )\n",
    "    \n",
    "    df = trial_component_analytics.dataframe()\n",
    "    print('is_best: ', str(df['is_best']))\n",
    "\n",
    "    result = False\n",
    "    if int(df['is_best']) > 0:\n",
    "        print('This model is the best ever!')\n",
    "        result = True\n",
    "    else:\n",
    "        print('This model is not so good!')\n",
    "    \n",
    "    return {\n",
    "        'statusCode'        : 200,\n",
    "        'result':result\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ./docker/lambda/Dockerfile\n",
    "\n",
    "# Define custom function directory\n",
    "ARG FUNCTION_DIR=\"/function\"\n",
    "\n",
    "FROM python:3.7-slim-buster\n",
    "    \n",
    "# Include global arg in this stage of the build\n",
    "ARG FUNCTION_DIR\n",
    "  \n",
    "RUN pip3 install --upgrade pip\n",
    "RUN pip3 install -qU boto3 pandas==0.25.3 sagemaker-experiments sagemaker awslambdaric\n",
    "\n",
    "# Copy function code\n",
    "RUN mkdir -p ${FUNCTION_DIR}\n",
    "COPY app/ ${FUNCTION_DIR}/\n",
    "\n",
    "# Set working directory to function root directory\n",
    "WORKDIR ${FUNCTION_DIR}\n",
    "\n",
    "\n",
    "ENTRYPOINT [ \"/usr/local/bin/python\", \"-m\", \"awslambdaric\" ]\n",
    "CMD [ \"app.handler\" ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ecr_repository_lambda = 'sf-lambda-' + user_name\n",
    "tag = ':latest'\n",
    "uri_suffix = 'amazonaws.com'\n",
    "lambda_repository_uri = '{}.dkr.ecr.{}.{}/{}'.format(account_id, region, uri_suffix, ecr_repository_lambda + tag)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create ECR repository and push docker image\n",
    "!docker build -t $ecr_repository_lambda docker/lambda\n",
    "!$(aws ecr get-login --region $region --registry-ids $account_id --no-include-email)\n",
    "!aws ecr create-repository --repository-name $ecr_repository_lambda\n",
    "!docker tag {ecr_repository_lambda + tag} $lambda_repository_uri\n",
    "!docker push $lambda_repository_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### コンテナイメージを使って AWS Lambda 関数を作成\n",
    "\n",
    "Lambda で使用するコンテナイメージができたら、コンテナイメージを使って Lambda 関数を作成します。ここからは、必要な権限を持つ IAM Role/Policy の作成と Lambda 関数の作成を行います。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import json\n",
    "\n",
    "def create_container_lambda_function(function_name, image_uri, policy_list=[], trust_service_list=[]):\n",
    "\n",
    "    timestamp = datetime.now(tz=JST).strftime('%Y%m%d%H%M')\n",
    "    lambda_function_name = function_name\n",
    "    lambda_inference_policy_name = lambda_function_name + '-policy-'+timestamp\n",
    "    lambda_inference_role_name = lambda_function_name + '-role-'+timestamp\n",
    "\n",
    "    inline_policy = {\n",
    "        \"Version\": \"2012-10-17\",\n",
    "        \"Statement\": [\n",
    "            {\n",
    "                \"Sid\": \"VisualEditor0\",\n",
    "                \"Effect\": \"Allow\",\n",
    "                \"Action\": \"sagemaker:Search\",\n",
    "                \"Resource\": \"*\"\n",
    "            },\n",
    "            {\n",
    "                'Effect': 'Allow',\n",
    "                'Action': 'logs:CreateLogGroup',\n",
    "                'Resource': f'arn:aws:logs:{region}:{account_id}:*'\n",
    "            },\n",
    "            {\n",
    "                'Effect': 'Allow',\n",
    "                'Action': [\n",
    "                    'logs:CreateLogStream',\n",
    "                    'logs:PutLogEvents'\n",
    "                ],\n",
    "                'Resource': [\n",
    "                    f'arn:aws:logs:{region}:{account_id}:log-group:/aws/lambda/{lambda_function_name}:*'\n",
    "                ]\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "\n",
    "    response = iam_client.create_policy(\n",
    "        PolicyName=lambda_inference_policy_name,\n",
    "        PolicyDocument=json.dumps(inline_policy),\n",
    "    )\n",
    "\n",
    "    policy_arn = response['Policy']['Arn']\n",
    "\n",
    "    service_list = [\"lambda.amazonaws.com\"]\n",
    "    for t in trust_service_list:\n",
    "        service_list.append(t)\n",
    "    \n",
    "    assume_role_policy = {\n",
    "      \"Version\": \"2012-10-17\",\n",
    "      \"Statement\": [{\"Sid\": \"\",\"Effect\": \"Allow\",\"Principal\": {\"Service\":service_list},\"Action\": \"sts:AssumeRole\"}]\n",
    "    }\n",
    "    response = iam_client.create_role(\n",
    "        Path = '/service-role/',\n",
    "        RoleName = lambda_inference_role_name,\n",
    "        AssumeRolePolicyDocument = json.dumps(assume_role_policy),\n",
    "        MaxSessionDuration=3600*12 # 12 hours\n",
    "    )\n",
    "    \n",
    "    lambda_role_arn = response['Role']['Arn']\n",
    "    lambda_role_name = response['Role']['RoleName']\n",
    "    \n",
    "    response = iam_client.attach_role_policy(\n",
    "        RoleName=lambda_inference_role_name,\n",
    "        PolicyArn=policy_arn\n",
    "    )\n",
    "    \n",
    "    for p in policy_list:\n",
    "        arn = 'arn:aws:iam::aws:policy/' + p\n",
    "        response = iam_client.attach_role_policy(\n",
    "            RoleName=lambda_inference_role_name,\n",
    "            PolicyArn=arn\n",
    "        )\n",
    "    \n",
    "    sleep(20) # wait until IAM is created\n",
    "    \n",
    "    response = lambda_client.create_function(\n",
    "        FunctionName=function_name,\n",
    "        Role=lambda_role_arn,\n",
    "        Code={\n",
    "            'ImageUri':image_uri\n",
    "        },\n",
    "        Timeout=60*5, # 5 minutes\n",
    "        MemorySize=128, # 128 MB\n",
    "        Publish=True,\n",
    "        PackageType='Image',\n",
    "    )\n",
    "    \n",
    "    return lambda_role_name, policy_arn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lambda_function_name_eval = 'evaluate-model-' + user_name\n",
    "lambda_role, lambda_policy = create_container_lambda_function(lambda_function_name_eval, lambda_repository_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "既存の Lambda 関数で使用するコンテナイメージを更新する場合は、以下のセルのコメントアウトを外して実行してください。新しいコンテナイメージが反映\n",
    "されるまで2分ほどお待ちください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# response = lambda_client.update_function_code(\n",
    "#         FunctionName=lambda_function_name_eval,\n",
    "#         ImageUri=lambda_repository_uri\n",
    "#     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LambdaStep の作成\n",
    "\n",
    "作成した Lambda 関数を使って `LambdaStep` を作成します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stepfunctions.steps.states import Retry\n",
    "lambda_step = stepfunctions.steps.compute.LambdaStep(\n",
    "    \"Query Evaluation Results\",\n",
    "    parameters={\n",
    "        \"FunctionName\": lambda_function_name_eval,\n",
    "        \"Payload\": {\n",
    "            \"experiment-name\": execution_input[\"ExperimentName\"],\n",
    "             \"evaluation-job-name\": execution_input[\"EvaluationProcessingJobName\"],\n",
    "        },\n",
    "    },\n",
    ")\n",
    "lambda_step.add_retry(\n",
    "    Retry(error_equals=[\"States.TaskFailed\"], interval_seconds=15, max_attempts=2, backoff_rate=4.0)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fail 状態の作成\n",
    "いずれかのステップが失敗したときにワークフローが失敗だとわかるように `Fail` 状態を作成します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "failed_state_sagemaker_processing_failure = stepfunctions.steps.states.Fail(\n",
    "    \"ML Workflow failed\", cause=\"SageMakerProcessingJobFailed\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ワークフローの中のエラーハンドリングを追加\n",
    "\n",
    "エラーハンドリングのために [Catch Block](https://aws-step-functions-data-science-sdk.readthedocs.io/en/stable/states.html#stepfunctions.steps.states.Catch) を使用します。もし いずれかの Step が失敗したら、`Fail` 状態に遷移します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "catch_state_processing = stepfunctions.steps.states.Catch(\n",
    "    error_equals=[\"States.TaskFailed\"],\n",
    "    next_step=failed_state_sagemaker_processing_failure,\n",
    ")\n",
    "\n",
    "processing_step.add_catch(catch_state_processing)\n",
    "processing_evaluation_step.add_catch(catch_state_processing)\n",
    "training_step.add_catch(catch_state_processing)\n",
    "lambda_step.add_catch(catch_state_processing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Workflow` の作成と実行\n",
    "\n",
    "ここまでで Step Functions のワークフローを作成する準備が完了しました。それでは、ワークフローを作成して実行してみましょう。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chain を使って各 Step を連結してワークフローを作成します。既存のワークフローを変更する場合は、update() を実行します。ログに ERROR が表示された場合は、以下のセルを再度実行してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow_graph = Chain([processing_step, training_step, processing_evaluation_step, lambda_step])\n",
    "\n",
    "branching_workflow = Workflow(\n",
    "    name=\"SageMakerProcessingWorkflow-\" + user_name,\n",
    "    definition=workflow_graph,\n",
    "    role=workflow_execution_role,\n",
    ")\n",
    "\n",
    "branching_workflow.create()\n",
    "branching_workflow.update(workflow_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ワークフローを複数回実行したい場合は、以下のセルから `branching_workflow.execute()` のセルまでの 3つのセルを 1セットとして実行してください。SageMaker のジョブ名は重複させることができないため、timestamp を使ってジョブ名を生成してからワークフローを実行する必要があります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timestamp = datetime.now(tz=JST).strftime('%Y%m%d-%H%M%S')\n",
    "\n",
    "# Generate unique names for Pre-Processing Job, Training Job, and Model Evaluation Job for the Step Functions Workflow\n",
    "training_job_name = \"scikit-learn-training-{}-{}\".format(\n",
    "    user_name, timestamp\n",
    ")  # Each Training Job requires a unique name\n",
    "preprocessing_job_name = \"scikit-learn-sm-preprocessing-{}-{}\".format(\n",
    "    user_name, timestamp\n",
    ")  # Each Preprocessing job requires a unique name,\n",
    "evaluation_job_name = \"scikit-learn-sm-evaluation-{}-{}\".format(\n",
    "    user_name, timestamp\n",
    ")  # Each Evaluation Job requires a unique name\n",
    "timestamp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Processing ジョブの出力を保存する S3 パスを作成します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 前処理ジョブ用\n",
    "s3_bucket_base_uri = \"{}{}\".format(\"s3://\", bucket)\n",
    "# output_data = \"{}/{}/{}-{}\".format(s3_bucket_base_uri, user_name, \"data/sklearn_processing/output\", timestamp)\n",
    "output_data = \"{}/{}/{}\".format(s3_bucket_base_uri, preprocessing_job_name, \"data/sklearn_processing/output\")\n",
    "preprocessed_training_data = \"{}/{}\".format(output_data, \"train_data\")\n",
    "\n",
    "# モデル評価ジョブ用\n",
    "preprocessed_testing_data = \"{}/{}\".format(output_data, \"test_data\")\n",
    "model_data_s3_uri = \"{}/{}/{}\".format(s3_bucket_base_uri, training_job_name, \"output/model.tar.gz\")\n",
    "output_model_evaluation_s3_uri = \"{}/{}/{}\".format(\n",
    "    s3_bucket_base_uri, training_job_name, \"evaluation\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "パラメタを指定して、ワークフローを実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute workflow\n",
    "execution = branching_workflow.execute(\n",
    "    inputs={\n",
    "        \"PreprocessingJobName\": preprocessing_job_name,  # Each pre processing job (SageMaker processing job) requires a unique name,\n",
    "        \"PreprocessingInputData\": input_data,\n",
    "        \"PreprocessingOutputDataTrain\": output_data+'/train_data',\n",
    "        \"PreprocessingOutputDataTest\": output_data+'/test_data',\n",
    "        \"TrainingJobName\": training_job_name,  # Each Sagemaker Training job requires a unique name,\n",
    "        \"TrainingParameters\": {\n",
    "                                     \"sagemaker_program\": \"train.py\",\n",
    "                                     \"sagemaker_submit_directory\":  train_code,\n",
    "                                      \"c\": '1.1'\n",
    "        },\n",
    "        \"TrainingOutputModel\": model_data_s3_uri,\n",
    "        \"ExperimentName\": experiment_evaluate.experiment_name,\n",
    "        \"EvaluationScriptPath\": input_evaluation_code,\n",
    "        \"EvaluationProcessingJobName\": evaluation_job_name,  # Each SageMaker processing job requires a unique name,\n",
    "        \"EvaluationProcessingOutput\": output_model_evaluation_s3_uri,\n",
    "        \"EvaluationExperimentArgs\": ['--experiment-name', experiment_evaluate.experiment_name]\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下のセルを実行することで、Workflow の進行状況がわかります。実行開始から12分程度で完了します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution.render_progress()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Workflow の実行が完了したら、Experiment の中をのぞいてみましょう。まずは Experiment のデータを ExperimentAnalytics を使って読み出します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_expression = {\n",
    "    \"Filters\":[\n",
    "        {\n",
    "            \"Name\": \"TrialComponentName\",\n",
    "            \"Operator\": \"Contains\",\n",
    "            \"Value\": evaluation_job_name,\n",
    "        }\n",
    "    ],\n",
    "}\n",
    "\n",
    "trial_component_analytics = ExperimentAnalytics(\n",
    "    experiment_name=experiment_evaluate.experiment_name,\n",
    "    sort_by=\"parameters.accuracy\",\n",
    "#     search_expression=search_expression,\n",
    "#     sort_by=\"metrics.acc.max\",\n",
    "#     sort_order=\"Ascending\",# Ascending or Descending\n",
    "#     metric_names=['metric1', 'metric2'],\n",
    "#     parameter_names=['accuracy', 'roc_auc'],\n",
    "    input_artifact_names=[]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "読み出したデータを DataFrame 形式に変換して表示します。accuracy や roc_auc が記録されていることがわかります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = trial_component_analytics.dataframe()\n",
    "pd.set_option('display.max_columns', None)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "表示が省略されて見にくいので、DataFrame を HTML ファイルに変換して保存します。以下のセルを実行すると、このノートブックと同じパスに experiments.html というファイルが作成されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('colheader_justify', 'center') \n",
    "\n",
    "html_string = '''\n",
    "<html>\n",
    "  <head><title>Experiments</title></head>\n",
    "  <body>\n",
    "    {table}\n",
    "  </body>\n",
    "</html>.\n",
    "'''\n",
    "\n",
    "# OUTPUT AN HTML FILE\n",
    "with open('experiments.html', 'w') as f:\n",
    "    f.write(html_string.format(table=df.to_html()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ノートブックインスタンスでこのノートブックを実行している場合、以下のセルを実行して experiments.html の内容を確認できます。SageMaker Studio を使用している場合は、直接 html ファイルをクリックして確認してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('experiments.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ワークフローの出力を確認\n",
    "\n",
    "Amazon S3 から `evaluation.json` を取得して確認します。ここにはモデルの評価レポートが書かれています。なお、以下のセルは Step Functions でワークフローの実行が完了してから（`evaluation.json` が出力されてから）実行してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow_execution_output_json = execution.get_output(wait=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Downloader\n",
    "import json\n",
    "\n",
    "evaluation_s3_uri = output_model_evaluation_s3_uri + '/evaluation.json'\n",
    "evaluation_output = S3Downloader.read_file(evaluation_s3_uri)\n",
    "evaluation_output_dict = json.loads(evaluation_output)\n",
    "print(json.dumps(evaluation_output_dict, sort_keys=True, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step Functions Workflow の実行のみ行う場合\n",
    "\n",
    "ここまでで、Step Functions Data Science SDK を使って Workflow を作成し、実行するところまでご紹介しました。実際に使用する際は、作成済みの Workflow をパラメタを指定して実行する部分を繰り返すことになります。ここでは、作成済みの Workflow を実行する部分のみを抜き出してご紹介します。\n",
    "\n",
    "Workflow の ARN を使って既存の Workflow を読み出します。ARN はわからないけれど Workflow 名はわかる、という場合は以下のセルを実行することで ARN を取得することができます。（`workflow_name` に Workflow 名を入れてください）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workflow_name = 'SageMakerProcessingWorkflow-' + user_name\n",
    "workflow_list = Workflow.list_workflows()\n",
    "workflow_arn = [d['stateMachineArn'] for d in workflow_list  if d['name']==workflow_name][0]\n",
    "workflow_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Workflow を読み出して `execute` で実行します。実行する際にパラメタを指定することができます。SageMaker ジョブはユニークな名前である必要があるため、このノートブックでは datetime を使ってジョブ名を生成しています。学習に使用するコードを変更する場合は、使用したいスクリプト（source.tar.gz）をアップロードしてある S3 パスを以下のセルの `train_code` に設定し、コメントアウトを解除してから実行してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "from dateutil import tz\n",
    "from stepfunctions.workflow import Workflow\n",
    "\n",
    "JST = tz.gettz('Asia/Tokyo')\n",
    "region = sagemaker_session.boto_region_name\n",
    "input_data = \"s3://sagemaker-sample-data-{}/processing/census/census-income.csv\".format(region)\n",
    "# train_code = 's3://xxx/code/source.tar.gz'\n",
    "\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "\n",
    "timestamp = datetime.now(tz=JST).strftime('%Y%m%d-%H%M%S')\n",
    "\n",
    "# Generate unique names for Pre-Processing Job, Training Job, and Model Evaluation Job for the Step Functions Workflow\n",
    "training_job_name = \"scikit-learn-training-{}-{}\".format(\n",
    "    user_name, timestamp\n",
    ")  # Each Training Job requires a unique name\n",
    "preprocessing_job_name = \"scikit-learn-sm-preprocessing-{}-{}\".format(\n",
    "    user_name, timestamp\n",
    ")  # Each Preprocessing job requires a unique name,\n",
    "evaluation_job_name = \"scikit-learn-sm-evaluation-{}-{}\".format(\n",
    "    user_name, timestamp\n",
    ")  # Each Evaluation Job requires a unique name\n",
    "\n",
    "# 前処理ジョブ用\n",
    "s3_bucket_base_uri = \"{}{}\".format(\"s3://\", bucket)\n",
    "output_data = \"{}/{}/{}\".format(s3_bucket_base_uri, preprocessing_job_name, \"data/sklearn_processing/output\")\n",
    "preprocessed_training_data = \"{}/{}\".format(output_data, \"train_data\")\n",
    "\n",
    "# モデル評価ジョブ用\n",
    "preprocessed_testing_data = \"{}/{}\".format(output_data, \"test_data\")\n",
    "model_data_s3_uri = \"{}/{}/{}\".format(s3_bucket_base_uri, training_job_name, \"output/model.tar.gz\")\n",
    "output_model_evaluation_s3_uri = \"{}/{}/{}\".format(\n",
    "    s3_bucket_base_uri, training_job_name, \"evaluation\"\n",
    ")\n",
    "\n",
    "existing_workflow = Workflow.attach(workflow_arn)\n",
    "execution = existing_workflow.execute(\n",
    "    inputs={\n",
    "        \"PreprocessingJobName\": preprocessing_job_name,  # Each pre processing job (SageMaker processing job) requires a unique name,\n",
    "        \"PreprocessingInputData\": input_data,\n",
    "        \"PreprocessingOutputDataTrain\": output_data+'/train_data',\n",
    "        \"PreprocessingOutputDataTest\": output_data+'/test_data',\n",
    "        \"TrainingJobName\": training_job_name,  # Each Sagemaker Training job requires a unique name,\n",
    "        \"TrainingParameters\": {\n",
    "                                     \"sagemaker_program\": \"train.py\",\n",
    "                                     \"sagemaker_submit_directory\":  train_code,\n",
    "                                      \"c\": '1.1'\n",
    "        },\n",
    "        \"TrainingOutputModel\": model_data_s3_uri,\n",
    "        \"ExperimentName\": experiment_evaluate.experiment_name,\n",
    "        \"EvaluationScriptPath\": input_evaluation_code,\n",
    "        \"EvaluationProcessingJobName\": evaluation_job_name,  # Each SageMaker processing job requires a unique name,\n",
    "        \"EvaluationProcessingOutput\": output_model_evaluation_s3_uri,\n",
    "        \"EvaluationExperimentArgs\": ['--experiment-name', experiment_evaluate.experiment_name]\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下のセルを実行することで、Workflow の進行状況がわかります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution.render_progress()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## リソースの削除\n",
    "\n",
    "このノートブックの実行が終わったら、不要なリソースを削除することを忘れないでください。このノートブックを最後まで実行してリソースの削除をしたら、ノートブックインスタンス、各種データを保存した S3 バケットも不要であれば削除してください。\n",
    "\n",
    "### Step Functions Workflow の削除"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "branching_workflow.delete()\n",
    "\n",
    "def detach_role_policies(role_name):\n",
    "    response = iam_client.list_attached_role_policies(\n",
    "        RoleName=role_name,\n",
    "    )\n",
    "    policies = response['AttachedPolicies']\n",
    "\n",
    "    for p in policies:\n",
    "        response = iam_client.detach_role_policy(\n",
    "            RoleName=role_name,\n",
    "            PolicyArn=p['PolicyArn']\n",
    "        )\n",
    "    \n",
    "detach_role_policies(step_functions_role_name)\n",
    "iam_client.delete_role(RoleName=step_functions_role_name)\n",
    "iam_client.delete_policy(PolicyArn=step_functions_policy_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiment の削除"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm = boto3.Session().client('sagemaker')\n",
    "def cleanup(experiment):\n",
    "    for trial_summary in experiment.list_trials():\n",
    "        trial = Trial.load(sagemaker_boto_client=sm, trial_name=trial_summary.trial_name)\n",
    "        for trial_component_summary in trial.list_trial_components():\n",
    "            tc = TrialComponent.load(\n",
    "                sagemaker_boto_client=sm,\n",
    "                trial_component_name=trial_component_summary.trial_component_name)\n",
    "            trial.remove_trial_component(tc)\n",
    "            try:\n",
    "                # comment out to keep trial components\n",
    "                tc.delete()\n",
    "            except:\n",
    "                # tc is associated with another trial\n",
    "                continue\n",
    "            # to prevent throttling\n",
    "            time.sleep(.5)\n",
    "        trial.delete()\n",
    "    experiment.delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleanup(experiment_evaluate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lambda 関数の削除"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lambda_role, lambda_policy \n",
    "\n",
    "detach_role_policies(lambda_role)\n",
    "iam_client.delete_role(RoleName=lambda_role)\n",
    "iam_client.delete_policy(PolicyArn=lambda_policy)\n",
    "    \n",
    "lambda_client.delete_function(FunctionName=lambda_function_name_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ECR リポジトリの削除"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ecr_client.delete_repository(\n",
    "    repositoryName=ecr_repository_lambda,\n",
    "    force=True\n",
    ")\n",
    "\n",
    "ecr_client.delete_repository(\n",
    "    repositoryName=ecr_repository,\n",
    "    force=True\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}