{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How to use Amazon Forecast\n", "\n", "Helps advanced users start with Amazon Forecast quickly. The demo notebook runs through a typical end to end usecase for a simple timeseries forecasting scenario. \n", "\n", "Prerequisites: \n", "[AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/installing.html) . \n", "\n", "For more informations about APIs, please check the [documentation](https://docs.aws.amazon.com/forecast/latest/dg/what-is-forecast.html)\n", "\n", "## Table Of Contents\n", "* [Setting up](#setup)\n", "* [Test Setup - Running first API](#hello)\n", "* [Forecasting Example with Amazon Forecast](#forecastingExample)\n", "\n", "**Read Every Cell FULLY before executing it**\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set up Preview SDK" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Configures your AWS CLI to now understand our up and coming service Amazon Forecast\n", "!aws configure add-model --service-model file://../sdk/forecastquery-2018-06-26.normal.json --service-name forecastquery\n", "!aws configure add-model --service-model file://../sdk/forecast-2018-06-26.normal.json --service-name forecast" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Prerequisites : 1 time install only, remove the comments to execute the lines.\n", "#!pip install boto3\n", "#!pip install pandas" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "from time import sleep\n", "import subprocess" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "session = boto3.Session(region_name='us-west-2') #us-east-1 is also supported\n", "\n", "forecast = session.client(service_name='forecast')\n", "forecastquery = session.client(service_name='forecastquery')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Test Setup \n", "Let's say Hi to the Amazon Forecast to interact with our Simple API ListRecipes. The API returns a list of the global recipes Forecast offers that you could potentially use as a part of your forecasting solution. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "{'RecipeNames': ['forecast_ARIMA',\n", " 'forecast_DEEP_AR',\n", " 'forecast_DEEP_AR_PLUS',\n", " 'forecast_ETS',\n", " 'forecast_MDN',\n", " 'forecast_MQRNN',\n", " 'forecast_NPTS',\n", " 'forecast_PROPHET',\n", " 'forecast_SQF'],\n", " 'ResponseMetadata': {'RequestId': '01ec50a4-e776-40b0-889c-66ed2f5213a2',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n", " 'date': 'Thu, 10 Jan 2019 16:25:09 GMT',\n", " 'x-amzn-requestid': '01ec50a4-e776-40b0-889c-66ed2f5213a2',\n", " 'content-length': '174',\n", " 'connection': 'keep-alive'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "forecast.list_recipes()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*If this ran successfully, kudos! If there are any errors at this point runing the following list_recipes, please contact us at the [AWS support forum](https://forums.aws.amazon.com/forum.jspa?forumID=327)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Forecasting with Amazon Forecast\n", "### Preparing your Data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In Amazon Forecast , a dataset is a collection of file(s) which contain data that is relevant for a forecasting task. A dataset must conform to a schema provided by Amazon Forecast. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this exercise, we use the individual household electric power consumption dataset. (Dua, D. and Karra Taniskidou, E. (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.) We aggregate the usage data hourly. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Data Type" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Amazon forecast can import data from Amazon S3. We first explore the data locally to see the fields" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
2014-01-01 01:00:0038.34991708126038client_12
02014-01-01 02:00:0033.5820895522388client_12
12014-01-01 03:00:0034.41127694859037client_12
22014-01-01 04:00:0039.800995024875625client_12
\n", "
" ], "text/plain": [ " 2014-01-01 01:00:00 38.34991708126038 client_12\n", "0 2014-01-01 02:00:00 33.5820895522388 client_12\n", "1 2014-01-01 03:00:00 34.41127694859037 client_12\n", "2 2014-01-01 04:00:00 39.800995024875625 client_12" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "df = pd.read_csv(\"../data/item-demand-time.csv\", dtype = object)\n", "df.head(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now upload the data to S3. But before doing that, go into your AWS Console, select S3 for the service and create a new bucket inside the `Oregon` or `us-west-2` region. Use that bucket name convention of `amazon-forecast-unique-value-data`. The name must be unique, if you get an error, just adjust until your name works, then update the `bucketName` cell below." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "s3 = session.client('s3')" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "accountId = boto3.client('sts').get_caller_identity().get('Account')" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "bucketName = 'amazon-forecast-ttsang-data'# Update the unique-value bit here.\n", "key=\"elec_data/item-demand-time.csv\"" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "s3.upload_file(Filename=\"../data/item-demand-time.csv\", Bucket=bucketName, Key=key)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'amazon-forecast-ttsang-data'" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "bucketName" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# One time setup only, uncomment the following command to create the role to provide to Amazon Forecast. \n", "# Save the generated role for all future calls to use for importing or exporting data. \n", "\n", "cmd = 'python ../setup_forecast_permissions.py '+bucketName\n", "p = subprocess.Popen(cmd.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "roleArn = 'arn:aws:iam::%s:role/amazonforecast'%accountId" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### CreateDataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "More details about `Domain` and dataset type can be found on the [documentation](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-domains-ds-types.html) . For this example, we are using [CUSTOM](https://docs.aws.amazon.com/forecast/latest/dg/custom-domain.html) domain with 3 required attributes `timestamp`, `target_value` and `item_id`. Also for your project name, update it to reflect your name in a lowercase format." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "DATASET_FREQUENCY = \"H\" \n", "TIMESTAMP_FORMAT = \"yyyy-MM-dd hh:mm:ss\"" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "project = 'ttsang_forecastdemo' # Replace this with a unique name here, make sure the entire name is < 30 characters.\n", "datasetName= project+'_ds'\n", "datasetGroupName= project +'_gp'\n", "s3DataPath = \"s3://\"+bucketName+\"/\"+key" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'ttsang_forecastdemo_ds'" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "datasetName" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "ename": "ResourceAlreadyExistsException", "evalue": "An error occurred (ResourceAlreadyExistsException) when calling the CreateDataset operation: Failed to create Dataset", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mResourceAlreadyExistsException\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 24\u001b[0m \u001b[0mDataFrequency\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mDATASET_FREQUENCY\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 25\u001b[0m \u001b[0mTimeStampFormat\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mTIMESTAMP_FORMAT\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m \u001b[0mSchema\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mschema\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 27\u001b[0m )\n", "\u001b[0;32m~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 318\u001b[0m \"%s() only accepts keyword arguments.\" % py_operation_name)\n\u001b[1;32m 319\u001b[0m \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 320\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 321\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 322\u001b[0m \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m 622\u001b[0m \u001b[0merror_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 623\u001b[0m \u001b[0merror_class\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 624\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0merror_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moperation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 625\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 626\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mResourceAlreadyExistsException\u001b[0m: An error occurred (ResourceAlreadyExistsException) when calling the CreateDataset operation: Failed to create Dataset" ] } ], "source": [ "# Specify the schema of your dataset here. Make sure the order of columns matches the raw data files.\n", "schema ={\n", " \"Attributes\":[\n", " {\n", " \"AttributeName\":\"timestamp\",\n", " \"AttributeType\":\"timestamp\"\n", " },\n", " {\n", " \"AttributeName\":\"target_value\",\n", " \"AttributeType\":\"float\"\n", " },\n", " {\n", " \"AttributeName\":\"item_id\",\n", " \"AttributeType\":\"string\"\n", " }\n", " ]\n", "}\n", "\n", "response=forecast.create_dataset(\n", " Domain=\"CUSTOM\",\n", " DatasetType='TARGET_TIME_SERIES',\n", " DataFormat='CSV',\n", " DatasetName=datasetName,\n", " DataFrequency=DATASET_FREQUENCY, \n", " TimeStampFormat=TIMESTAMP_FORMAT,\n", " Schema = schema\n", " )" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'DatasetName': 'ttsang_forecastdemo_ds',\n", " 'DatasetType': 'TARGET_TIME_SERIES',\n", " 'DataFormat': 'CSV',\n", " 'Domain': 'CUSTOM',\n", " 'ScheduleExpression': 'none',\n", " 'DatasetArn': 'arn:aws:forecast:us-west-2:073138737237:ds/ttsang_forecastdemo_ds',\n", " 'Status': 'ACTIVE',\n", " 'ResponseMetadata': {'RequestId': '0cd758aa-6473-43ed-9230-6efec4999397',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n", " 'date': 'Thu, 10 Jan 2019 16:53:51 GMT',\n", " 'x-amzn-requestid': '0cd758aa-6473-43ed-9230-6efec4999397',\n", " 'content-length': '239',\n", " 'connection': 'keep-alive'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "forecast.describe_dataset(DatasetName=datasetName)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "ename": "ResourceAlreadyExistsException", "evalue": "An error occurred (ResourceAlreadyExistsException) when calling the CreateDatasetGroup operation: Resource already exists", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mResourceAlreadyExistsException\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mforecast\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_dataset_group\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mDatasetGroupName\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdatasetGroupName\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mRoleArn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mroleArn\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mDatasetNames\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdatasetName\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", "\u001b[0;32m~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 318\u001b[0m \"%s() only accepts keyword arguments.\" % py_operation_name)\n\u001b[1;32m 319\u001b[0m \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 320\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 321\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 322\u001b[0m \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;32m~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m 622\u001b[0m \u001b[0merror_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 623\u001b[0m \u001b[0merror_class\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 624\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0merror_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moperation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 625\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 626\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mResourceAlreadyExistsException\u001b[0m: An error occurred (ResourceAlreadyExistsException) when calling the CreateDatasetGroup operation: Resource already exists" ] } ], "source": [ "forecast.create_dataset_group(DatasetGroupName=datasetGroupName,RoleArn=roleArn,DatasetNames=[datasetName])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you have an existing datasetgroup, you can update it" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'DatasetGroupName': 'ttsang_forecastdemo_gp',\n", " 'DatasetGroupArn': 'arn:aws:forecast:us-west-2:073138737237:dsgroup/ttsang_forecastdemo_gp',\n", " 'Datasets': ['ttsang_forecastdemo_ds'],\n", " 'RoleArn': 'arn:aws:iam::073138737237:role/amazonforecast',\n", " 'KMSKeyArn': 'arn:aws:kms:us-west-2:010754705615:key/8d09445a-efe1-4a36-b212-5845ae10c4ad',\n", " 'ResponseMetadata': {'RequestId': '8337c438-2911-4021-aab7-3f5342b6d701',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n", " 'date': 'Thu, 10 Jan 2019 16:53:57 GMT',\n", " 'x-amzn-requestid': '8337c438-2911-4021-aab7-3f5342b6d701',\n", " 'content-length': '322',\n", " 'connection': 'keep-alive'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "forecast.describe_dataset_group(DatasetGroupName=datasetGroupName)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Data Import Job\n", "Brings the data into Amazon Forecast system ready to forecast from raw data. " ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "ds_import_job_response=forecast.create_dataset_import_job(DatasetName=datasetName,Delimiter=',', DatasetGroupName =datasetGroupName ,S3Uri= s3DataPath)\n" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "62932d40\n" ] } ], "source": [ "ds_versionId=ds_import_job_response['VersionId']\n", "print(ds_versionId)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check the status of dataset, when the status change from **CREATING** to **ACTIVE**, we can continue to next steps. Depending on the data size. It can take 10 mins to be **ACTIVE**. This process will take 5 to 10 minutes." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "QUEUED\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "ACTIVE\n" ] } ], "source": [ "while True:\n", " dataImportStatus = forecast.describe_dataset_import_job(DatasetName=datasetName,VersionId=ds_versionId)['Status']\n", " print(dataImportStatus)\n", " if dataImportStatus != 'ACTIVE' and dataImportStatus != 'FAILED':\n", " sleep(30)\n", " else:\n", " break" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'DatasetArn': 'arn:aws:forecast:us-west-2:073138737237:ds/ttsang_forecastdemo_ds',\n", " 'DatasetName': 'ttsang_forecastdemo_ds',\n", " 'VersionId': '1be04189',\n", " 'Status': 'ACTIVE',\n", " 'FieldStatistics': {'date': {'Count': 26280,\n", " 'CountDistinct': 8396,\n", " 'CountNull': 0,\n", " 'Min': '2014-01-01T00:00:00Z',\n", " 'Max': '2015-01-01T00:00:00Z'},\n", " 'item': {'Count': 26280, 'CountDistinct': 3, 'CountNull': 0},\n", " 'target': {'Count': 26280,\n", " 'CountDistinct': 5059,\n", " 'CountNull': 0,\n", " 'Min': '0.0',\n", " 'Max': '212.27197346600326',\n", " 'Avg': 50.82350576202014,\n", " 'Stddev': 37.9125549309785}},\n", " 'ResponseMetadata': {'RequestId': '314772a8-7ecd-4b13-a437-8d64c3eda371',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n", " 'date': 'Wed, 09 Jan 2019 21:25:26 GMT',\n", " 'x-amzn-requestid': '314772a8-7ecd-4b13-a437-8d64c3eda371',\n", " 'content-length': '502',\n", " 'connection': 'keep-alive'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "forecast.describe_dataset_import_job(DatasetName=datasetName,VersionId=ds_versionId)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Recipe" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'RecipeNames': ['forecast_ARIMA',\n", " 'forecast_DEEP_AR',\n", " 'forecast_DEEP_AR_PLUS',\n", " 'forecast_ETS',\n", " 'forecast_MDN',\n", " 'forecast_MQRNN',\n", " 'forecast_NPTS',\n", " 'forecast_PROPHET',\n", " 'forecast_SQF'],\n", " 'ResponseMetadata': {'RequestId': '01e9798c-2f0f-4fd9-ae7e-dfbf8daae35b',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n", " 'date': 'Wed, 09 Jan 2019 21:25:43 GMT',\n", " 'x-amzn-requestid': '01e9798c-2f0f-4fd9-ae7e-dfbf8daae35b',\n", " 'content-length': '174',\n", " 'connection': 'keep-alive'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "recipesResponse=forecast.list_recipes()\n", "recipesResponse" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get details about each recipe." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "forecast.describe_recipe(RecipeName='forecast_MQRNN')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create Solution with customer forecast horizon" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Forecast horizon is how long in future the forecast should be predicting. For weekly data, a value of 12 means 1 weeks. Our example is hourly data, we try forecast the next day, so we can set to 24." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "predictorName= project+'_mqrnn'" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "forecastHorizon = 24" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "createPredictorResponse=forecast.create_predictor(RecipeName='forecast_MQRNN',DatasetGroupName= datasetGroupName ,PredictorName=predictorName, \n", " ForecastHorizon = forecastHorizon)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "predictorVerionId=createPredictorResponse['VersionId']" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'PredictorVersions': [{'PredictorName': 'ttsang_forecastdemo_mqrnn',\n", " 'VersionId': '23f6568f'}],\n", " 'ResponseMetadata': {'RequestId': '7625390d-fe08-4601-9392-5e474700574b',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n", " 'date': 'Wed, 09 Jan 2019 21:28:04 GMT',\n", " 'x-amzn-requestid': '7625390d-fe08-4601-9392-5e474700574b',\n", " 'content-length': '92',\n", " 'connection': 'keep-alive'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "forecast.list_predictor_versions(PredictorName=predictorName)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check the status of solutions, when the status change from **CREATING** to **ACTIVE**, we can continue to next steps. Depending on data size, model selection and hyper parameters，it can take 10 mins to more than one hour to be **ACTIVE**." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "ACTIVE\n" ] } ], "source": [ "while True:\n", " predictorStatus = forecast.describe_predictor(PredictorName=predictorName,VersionId=predictorVerionId)['Status']\n", " print(predictorStatus)\n", " if predictorStatus != 'ACTIVE' and predictorStatus != 'FAILED':\n", " sleep(30)\n", " else:\n", " break" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Get Error Metrics" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'ModelMetrics': {'MQRNN': {'Metrics': {'p10': '0.15023963544745242',\n", " 'p50': '0.3194856145135812',\n", " 'p90': '0.2858658044059431',\n", " 'rmse': '20.818507895020403'},\n", " 'MetricsByBucket': []}},\n", " 'ResponseMetadata': {'RequestId': 'ee667b3d-700a-489c-a997-83500ef3b094',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n", " 'date': 'Wed, 09 Jan 2019 21:56:53 GMT',\n", " 'x-amzn-requestid': 'ee667b3d-700a-489c-a997-83500ef3b094',\n", " 'content-length': '171',\n", " 'connection': 'keep-alive'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "forecastquery.get_accuracy_metrics(PredictorName=predictorName)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deploy Predictor" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'PredictorName': 'ttsang_forecastdemo_mqrnn',\n", " 'VersionId': '23f6568f',\n", " 'PredictorArn': 'arn:aws:forecast:us-west-2:073138737237:predictor/ttsang_forecastdemo_mqrnn',\n", " 'ResponseMetadata': {'RequestId': '34751dd8-a0f7-4b6e-b2e0-6dc262938294',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n", " 'date': 'Wed, 09 Jan 2019 21:59:56 GMT',\n", " 'x-amzn-requestid': '34751dd8-a0f7-4b6e-b2e0-6dc262938294',\n", " 'content-length': '161',\n", " 'connection': 'keep-alive'},\n", " 'RetryAttempts': 0}}" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "forecast.deploy_predictor(PredictorName=predictorName)" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'PredictorNames': ['ttsang_forecastdemo_mqrnn'], 'ResponseMetadata': {'RequestId': '92c44eb3-a338-4fac-9893-8370c5ef1db4', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1', 'date': 'Wed, 09 Jan 2019 22:00:06 GMT', 'x-amzn-requestid': '92c44eb3-a338-4fac-9893-8370c5ef1db4', 'content-length': '48', 'connection': 'keep-alive'}, 'RetryAttempts': 0}}\n" ] } ], "source": [ "deployedPredictorsResponse=forecast.list_deployed_predictors()\n", "print(deployedPredictorsResponse)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Please note that the following cell can also take 10 minutes or more to be fully operational. There's no output here, but that is fine as long as the * is there." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "ACTIVE\n", "ACTIVE\n" ] } ], "source": [ "while True:\n", " deployedPredictorStatus = forecast.describe_deployed_predictor(PredictorName=predictorName)['Status']\n", " print(deployedPredictorStatus)\n", " if deployedPredictorStatus != 'ACTIVE' and deployedPredictorStatus != 'FAILED':\n", " sleep(30)\n", " else:\n", " break\n", "print(deployedPredictorStatus)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Get Forecast" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When the solution is deployed and forecast results are ready, you can view them. " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "ename": "NameError", "evalue": "name 'forecastquery' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m forecastResponse = forecastquery.get_forecast(\n\u001b[0m\u001b[1;32m 2\u001b[0m \u001b[0mPredictorName\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpredictorName\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0mInterval\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"hour\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mFilters\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m\"item_id\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\"client_12\"\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m )\n", "\u001b[0;31mNameError\u001b[0m: name 'forecastquery' is not defined" ] } ], "source": [ "forecastResponse = forecastquery.get_forecast(\n", " PredictorName=predictorName,\n", " Interval=\"hour\",\n", " Filters={\"item_id\":\"client_12\"}\n", ")\n", "print(forecastResponse)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Export Forecast" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### You can batch export forecast to s3 bucket. To do so an role with s3 put access is needed, but this has already been created." ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "forecastInfoList= forecast.list_forecasts(PredictorName=predictorName)['ForecastInfoList']\n", "forecastId= forecastInfoList[0]['ForecastId']" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [], "source": [ "outputPath=\"s3://\"+bucketName+\"/output\"" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "forecastExportResponse = forecast.create_forecast_export_job(ForecastId=forecastId, OutputPath={\"S3Uri\": outputPath,\"RoleArn\":roleArn})" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "forecastExportJobId = forecastExportResponse['ForecastExportJobId']" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "CREATING\n", "ACTIVE\n" ] } ], "source": [ "while True:\n", " forecastExportStatus = forecast.describe_forecast_export_job(ForecastExportJobId=forecastExportJobId)['Status']\n", " print(forecastExportStatus)\n", " if forecastExportStatus != 'ACTIVE' and forecastExportStatus != 'FAILED':\n", " sleep(30)\n", " else:\n", " break" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check s3 bucket for results" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'ResponseMetadata': {'RequestId': 'B1CEB8078FE73E47',\n", " 'HostId': 'mWi1Nj90oPNfh5mB2wOid/+GSy2VfypGo5AcB/tzUWoTGQsd1YDxE1oakuPNtpVdGHX73Qu+/8w=',\n", " 'HTTPStatusCode': 200,\n", " 'HTTPHeaders': {'x-amz-id-2': 'mWi1Nj90oPNfh5mB2wOid/+GSy2VfypGo5AcB/tzUWoTGQsd1YDxE1oakuPNtpVdGHX73Qu+/8w=',\n", " 'x-amz-request-id': 'B1CEB8078FE73E47',\n", " 'date': 'Wed, 09 Jan 2019 22:52:51 GMT',\n", " 'x-amz-bucket-region': 'us-west-2',\n", " 'content-type': 'application/xml',\n", " 'transfer-encoding': 'chunked',\n", " 'server': 'AmazonS3'},\n", " 'RetryAttempts': 0},\n", " 'IsTruncated': False,\n", " 'Marker': '',\n", " 'Contents': [{'Key': 'output/_SUCCESS',\n", " 'LastModified': datetime.datetime(2019, 1, 9, 22, 22, 44, tzinfo=tzlocal()),\n", " 'ETag': '\"d41d8cd98f00b204e9800998ecf8427e\"',\n", " 'Size': 0,\n", " 'StorageClass': 'STANDARD',\n", " 'Owner': {'DisplayName': 'hackathonq12019',\n", " 'ID': '12046a474b3406052ef6ad4a28a7a1a7598ad764487e3a1772a60c8b6ce1a984'}},\n", " {'Key': 'output/part-00000-dfd137a3-af02-43fc-ba30-6eb8829ffa71-c000.csv',\n", " 'LastModified': datetime.datetime(2019, 1, 9, 22, 22, 44, tzinfo=tzlocal()),\n", " 'ETag': '\"61e33399a4bc0fea3353508ad206f37c\"',\n", " 'Size': 6365,\n", " 'StorageClass': 'STANDARD',\n", " 'Owner': {'DisplayName': 'hackathonq12019',\n", " 'ID': '12046a474b3406052ef6ad4a28a7a1a7598ad764487e3a1772a60c8b6ce1a984'}}],\n", " 'Name': 'amazon-forecast-ttsang-data',\n", " 'Prefix': 'output',\n", " 'MaxKeys': 1000,\n", " 'EncodingType': 'url'}" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.list_objects(Bucket=bucketName,Prefix=\"output\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Cleanup\n", "\n", "While Forecast is in preview there are no charges for using it, but to future proof this work below are the instructions to cleanup your work space." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Delete Deployed Predictor \n", "forecast.delete_deployed_predictor(PredictorName=predictorName)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Delete the Predictor: \n", "forecast.delete_predictor(PredictorName=predictorName)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Delete Import\n", "forecast.delete_dataset_import(DatasetName=datasetName)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Delete Dataset Group\n", "forecast.delete_dataset_group(DatasetGroupName=datasetGroupName)" ] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 2 }