{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Walkthrough for S3-helper function \n", "

\n", " Forget where you put some sensitive information on your cloud database?\n", "
Tired of downloading files from s3 only to preview them in excel, or textpad?\n", "
Want to touch and reshape data but feel it's caged off from you?\n", "
Look no further, your s3 problems are solved!\n", "

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Table of Contents \n", "
\n", " This iPython notebook gives a walkthrough of several handy functions from the s3 module.\n", "
With best intentions, these functions mirror the use of standard libraries, while empolying the backend of popular open source projects.\n", "

The notebook highlights 5 functions to:\n", "
\n", "\n", "1. List files (with wildcard) in a s3 bucket/key using ls()\n", "2. Read files into a string using open() \n", "3. Read csv and json files on s3 into Pandas dataframes using read_csv() and read_json\n", "4. Write csv and json files from Pandas dataframes to s3 using to_csv() and to_json()\n", "5. Write local files to s3 using write()\n", "6. Saving and Loading Scikit-Learn classifiers \n", "7. Moving files to new buckets and keys using mv()" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import os\n", "import s3 as s3\n", "from s3 import BUCKET_NAME" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "BUCKET_NAME is a constant, and can be changed in constants.py" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "'dev-rsaporta'" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "BUCKET_NAME" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Listing files in a S3 bucket and key using ls( ) \n", "

\n", " s3.look will list all the files and directories in a bucket/key akin to os.listdir()\n", "
see the code\n", "

The search_key (str) argument can contain an explicit bucket name by including \"s3://\"...\n", "

" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "ROOT = 's3://dev-rsaporta/DataAnalyticsDept/marketShare/apple/raw_storage/{}'" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "['DataAnalyticsDept/marketShare/apple/raw_storage/2015/08/S2_80027963_0915_AU.txt',\n", " 'DataAnalyticsDept/marketShare/apple/raw_storage/2015/08/S2_80027963_0915_CA.txt',\n", " 'DataAnalyticsDept/marketShare/apple/raw_storage/2015/08/S2_80027963_0915_CH.txt']" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.ls(ROOT.format('2015/08/*.txt'))[:3]" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "['DataAnalyticsDept/marketShare/apple/raw_storage/2015/07/S2_80027963_0815_AU.txt',\n", " 'DataAnalyticsDept/marketShare/apple/raw_storage/2015/07/S2_80027963_0815_CA.txt',\n", " 'DataAnalyticsDept/marketShare/apple/raw_storage/2015/07/S2_80027963_0815_CH.txt']" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.ls('s3://dev-rsaporta/DataAnalyticsDept/marketShare/apple/raw_storage')[:3]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Otherwise, the BUCKET_NAME variable is imported from constants.py." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "['DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v1/TheOrchard_201306.csv',\n", " 'DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v1/TheOrchard_201307.csv',\n", " 'DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v1/TheOrchard_201308.csv']" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.ls('DataAnalyticsDept/marketShare/deezer/raw_storage')[:3]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "s3.look also supports regex-like wildcard patterns exactly like glob.glob()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "['s3://dev-rsaporta/DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v3/102015summary.csv',\n", " 's3://dev-rsaporta/DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v3/112015summary.csv',\n", " 's3://dev-rsaporta/DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v3/122015summary.csv']" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.ls('DataAnalyticsDept/marketShare/deezer/*/data_in_v3/*.csv', include_bucket=True)[:3]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With a programmatic method of getting s3 file paths, we can start doing some cools stuff.\n", "
top" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Read files in s3 with open() \n", "This function as akin to Python's native \n", "
 Open(filename, 'r') as f: \n", "
   f.read()\n", "
see the code\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "'Apple Music\\r\\nCover Sheet\\r\\nPeriod\\t08/30/2015-09/26/2015\\r\\nCurrency\\tAUD\\r\\n\\r\\n\\tAustralia\\r\\nTotal Royalty Amount\\t3.66\\r\\nAverage Effective Per Play Rate\\t.024348\\r\\n\\r\\nINDIVIDUAL MONTHLY PAID\\t\\r\\nLabel Royalty Bearing Plays\\t-\\r\\n\\t\\r\\nPercentage of Revenue\\t58\\r\\nRevenue Share Amount\\t-\\r\\n\\t\\r\\nPer Subscriber Daily Minimum\\t.210666667\\r\\nPer Subscriber Minimum Amount\\t-\\r\\n\\t\\r\\nRoyalty Amount\\t-\\r\\nEffective Per Play Rate\\t-\\r\\n\\t\\r\\nFAMILY MONTHLY PAID\\t\\r\\nLabel Royalty Bearing Plays\\t-\\r\\n\\t\\r\\nPercentage of Revenue\\t58\\r\\nRevenue Share Amount\\t-\\r\\n\\t\\r\\nPer Subscriber Daily Minimum\\t.316\\r\\nPer Subscriber Minimum Amount\\t-\\r\\n\\t\\r\\nRoyalty Amount\\t-\\r\\nEffective Per Play Rate\\t-\\r\\n\\t\\r\\nINDIVIDUAL MONTHLY TRIAL\\t\\r\\nLabel Royalty Bearing Plays\\t737\\r\\n\\t\\r\\nPer Play Rate\\t.00218\\r\\nPer Play Amount\\t1.61\\r\\n\\t\\r\\nRoyalty Amount\\t1.61\\r\\nEffective Per Play Rate\\t.002185\\r\\n\\t\\r\\nFAMILY MONTHLY TRIAL\\t\\r\\nLabel Royalty Bearing Plays\\t107\\r\\n\\t\\r\\nPer Play Rate\\t.00218\\r\\nPer Play Amount\\t0.23\\r\\n\\t\\r\\nRoyalty Amount\\t0.23\\r\\nEffective Per Play Rate\\t.00215\\r\\n\\t\\r\\nINDIRECT INDIVIDUAL MONTHLY PAID\\t\\r\\nLabel Royalty Bearing Plays\\t-\\r\\n\\t\\r\\nPercentage of Revenue\\t58\\r\\nRevenue Share Amount\\t-\\r\\n\\t\\r\\nPer Subscriber Daily Minimum\\t.189666667\\r\\nPer Subscriber Minimum Amount\\t-\\r\\n\\t\\r\\nRoyalty Amount\\t-\\r\\nEffective Per Play Rate\\t-\\r\\n\\t\\r\\nINDIRECT INDIVIDUAL MONTHLY EXTENDED TRIAL\\t\\r\\nLabel Royalty Bearing Plays\\t36\\r\\n\\t\\r\\nPercentage of Revenue\\t58\\r\\nRevenue Share Amount\\t1.24\\r\\n\\t\\r\\nPer Subscriber Daily Minimum\\t.111\\r\\nPer Subscriber Minimum Amount\\t1.73\\r\\n\\t\\r\\nRoyalty Amount\\t1.73\\r\\nEffective Per Play Rate\\t.048056\\r\\n\\t\\r\\nINDIRECT INDIVIDUAL MONTHLY EXTENDED TRIAL (24 MONTH)\\t\\r\\nLabel Royalty Bearing Plays\\t2\\r\\n\\t\\r\\nPercentage of Revenue\\t58\\r\\nRevenue Share Amount\\t0.07\\r\\n\\t\\r\\nPer Subscriber Daily Minimum\\t.139\\r\\nPer Subscriber Minimum Amount\\t0.09\\r\\n\\t\\r\\nRoyalty Amount\\t0.09\\r\\nEffective Per Play Rate\\t.045\\r\\n\\t\\r\\n'" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "f = 's3://dev-rsaporta/DataAnalyticsDept/marketShare/apple/raw_storage/2015/09/S2_80027963_1015_AU.txt'\n", "\n", "s3.open(f, encoding='utf-8', bytes=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Without specifying the `bytes` argument, the function defaults to return a stream of bytes." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.open(f)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more structured data, we can leverage Panda's parsing engines...\n", "
top" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Read S3 files to memory with read_csv( ) and read_json( ) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " s3.read_csv and read_json are identical to their Pandas' ancestor and backbone.\n", "
Like s3.ls() the bucket name can be explicit if the argument includes \"s3://\"\n", "
see the code\n", "

" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MonthLabelOfferCountryDeezer RevenueCurrency EuroSubscribersNb StreamsCost in EuroCurrency Euro.1Cost in local currencyLocal currencyMarket Share
0201510The OrchardMODAE28.059747EUR0.01527613.865844EUR15.276001USD0.058569
1201510The OrchardMODAF0.196000EUR0.01770.160661EUR0.177000USD0.128821
2201510The OrchardMODAG0.090950EUR0.01100.099846EUR0.110000USD0.057173
3201510The OrchardMODAI0.000000EUR0.0440.039938EUR0.044000USD0.611111
4201510The OrchardMODAL2.430700EUR0.033913.077971EUR3.391000USD0.042610
\n", "
" ], "text/plain": [ " Month Label Offer Country Deezer Revenue Currency Euro \\\n", "0 201510 The Orchard MOD AE 28.059747 EUR \n", "1 201510 The Orchard MOD AF 0.196000 EUR \n", "2 201510 The Orchard MOD AG 0.090950 EUR \n", "3 201510 The Orchard MOD AI 0.000000 EUR \n", "4 201510 The Orchard MOD AL 2.430700 EUR \n", "\n", " Subscribers Nb Streams Cost in Euro Currency Euro.1 \\\n", "0 0.0 15276 13.865844 EUR \n", "1 0.0 177 0.160661 EUR \n", "2 0.0 110 0.099846 EUR \n", "3 0.0 44 0.039938 EUR \n", "4 0.0 3391 3.077971 EUR \n", "\n", " Cost in local currency Local currency Market Share \n", "0 15.276001 USD 0.058569 \n", "1 0.177000 USD 0.128821 \n", "2 0.110000 USD 0.057173 \n", "3 0.044000 USD 0.611111 \n", "4 3.391000 USD 0.042610 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = s3.read_csv('s3://dev-rsaporta/DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v3/102015summary.csv',sep=',')\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Bringing it all together\n", "The listdir feature of s3.look() can be leveraged to combine files into a larger dataframe:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1379 rows added from DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v3/102015summary.csv\n", "1378 rows added from DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v3/112015summary.csv\n", "1419 rows added from DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v3/122015summary.csv\n" ] }, { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MonthLabelOfferCountryDeezer RevenueCurrency EuroSubscribersNb StreamsCost in EuroCurrency Euro.1Cost in local currencyLocal currencyMarket Share
4171201512The OrchardThat \"Sick Deal\"DE1252.020000EUR231.017212.134840EUR12.134840EUR0.036713
4172201512The OrchardBMW FR Premium+ T&B 1 monthFR455.280000EUR84.0635.915795EUR5.915795EUR0.049219
4173201512The OrchardBMW IT Premium+ T&B 1 monthIT401.080000EUR74.0965.074873EUR5.074873EUR0.047928
4174201512The OrchardBMW UK Premium+ T&B 1 monthGB1321.547552EUR168.014023.908173EUR23.908173EUR0.068527
4175201512The OrchardBMW NL Premium+ T&B 1 monthNL308.940000EUR57.01309.383027EUR9.383027EUR0.115044
\n", "
" ], "text/plain": [ " Month Label Offer Country \\\n", "4171 201512 The Orchard That \"Sick Deal\" DE \n", "4172 201512 The Orchard BMW FR Premium+ T&B 1 month FR \n", "4173 201512 The Orchard BMW IT Premium+ T&B 1 month IT \n", "4174 201512 The Orchard BMW UK Premium+ T&B 1 month GB \n", "4175 201512 The Orchard BMW NL Premium+ T&B 1 month NL \n", "\n", " Deezer Revenue Currency Euro Subscribers Nb Streams Cost in Euro \\\n", "4171 1252.020000 EUR 231.0 172 12.134840 \n", "4172 455.280000 EUR 84.0 63 5.915795 \n", "4173 401.080000 EUR 74.0 96 5.074873 \n", "4174 1321.547552 EUR 168.0 140 23.908173 \n", "4175 308.940000 EUR 57.0 130 9.383027 \n", "\n", " Currency Euro.1 Cost in local currency Local currency Market Share \n", "4171 EUR 12.134840 EUR 0.036713 \n", "4172 EUR 5.915795 EUR 0.049219 \n", "4173 EUR 5.074873 EUR 0.047928 \n", "4174 EUR 23.908173 EUR 0.068527 \n", "4175 EUR 9.383027 EUR 0.115044 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = pd.DataFrame()\n", "index = 0\n", "\n", "for file in s3.ls('DataAnalyticsDept/marketShare/deezer/raw_storage/data_in_v3/*.csv'):\n", " \n", " # read the file in\n", " df_ghost = s3.read_csv(file, sep=',')\n", " df_ghost.index += index\n", "\n", " # do something\n", " df_ghost['Offer'].replace({'BMW DE Premium+ T&B 1 month': 'That \"Sick Deal\"'}, inplace=True)\n", " \n", " # add to the master dataframe\n", " df = df.append(df_ghost)\n", " \n", " # cook the books\n", " new_cols = len(df_ghost)\n", " index += new_cols\n", " print(\"{} rows added from {}\".format(new_cols,file))\n", "\n", "df.tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After processing and normalizing the data, we may want to upload this new file to s3.\n", "
top" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Write DataFrames to S3 with to_csv( ) and to_json( ) \n", "

\n", " s3.read_csv and read_json are almost identical to their Pandas ancestor and backbone.\n", "
The difference is that s3.to_csv takes the dataframe as an argument, rather then operates as a 'self' function.\n", "
see the code\n", "

" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "'dev-rsaporta/DataAnalyticsDept/one_offs/demo/test.csv.gz'" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3_location = os.path.join(BUCKET_NAME, 'DataAnalyticsDept/one_offs/demo/test.csv.gz')\n", "s3_location" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "\"File uploaded to 'dev-rsaporta/DataAnalyticsDept/one_offs/demo/test.csv.gz'\"" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.to_csv(df, s3_location, sep='\\t',\n", " index=False, compression='gzip')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "top" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Write local files to S3 with disk_2_s3( ) \n", "This is a fairly tyical workflow:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# create a file\n", "df.to_csv('test.csv', index=False)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "\"'test.csv' loaded to 's3://dev-rsaporta/DataAnalyticsDept/one_offs/demo/test.csv.gz'\"" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.disk_2_s3(file='test.csv',\n", " s3_path=s3_location)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# purge it!\n", "os.remove('test.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Saving and Loading Scikit-Learn Classifiers \n", "If you're into machine learning on the cloud, you're in luck!" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "from sklearn.linear_model import LassoCV" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "for the example let's just use a vanilla untrained Lasso Model" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "LassoCV(alphas=None, copy_X=True, cv=None, eps=0.001, fit_intercept=True,\n", " max_iter=1000, n_alphas=100, n_jobs=1, normalize=False, positive=False,\n", " precompute='auto', random_state=None, selection='cyclic', tol=0.0001,\n", " verbose=False)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf = LassoCV()\n", "clf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is where we'd train and evaluate the model..." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# clf.fit(X, y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once we're happy with the performance, we can persist the model as a pickle file." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "\"'clf.pkl' loaded to 's3://dev-rsaporta/DataAnalyticsDept/clf.pkl'\"" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.dump_clf(clf, 's3://dev-rsaporta/DataAnalyticsDept/clf.pkl')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And re-use it when the time is right!" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "LassoCV(alphas=None, copy_X=True, cv=None, eps=0.001, fit_intercept=True,\n", " max_iter=1000, n_alphas=100, n_jobs=1, normalize=False, positive=False,\n", " precompute='auto', random_state=None, selection='cyclic', tol=0.0001,\n", " verbose=False)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.load_clf('s3://dev-rsaporta/DataAnalyticsDept/clf.pkl')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "top" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Movin' Files between buckets and keys \n", "In the interest of good file-keeping let's move our saved classifier to it's own special folder (key).
\n", "To make a copy, set clear to false." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "{'CopyObjectResult': {'ETag': '\"3ff2be881324c5390dbf03bf77d28318\"',\n", " 'LastModified': datetime.datetime(2017, 2, 1, 16, 17, 12, tzinfo=tzutc())},\n", " 'ResponseMetadata': {'HTTPStatusCode': 200,\n", " 'HostId': 'rYv/t9B+c0+fNLCndeOjvs1HYwMIODCW6q86OXQdg6MzKn1k4zKrucZYuURso+SxsPzF7W9QP6M=',\n", " 'RequestId': 'B6749BBF94FC37CE'}}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.mv(old_path='s3://dev-rsaporta/DataAnalyticsDept/clf.pkl',\n", " new_path='s3://dev-rsaporta/DataAnalyticsDept/one_offs/sklearn_clf/clf.pkl',\n", " keep=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "to move the file (and delete the old instance) don't set clear." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "{'ResponseMetadata': {'HTTPStatusCode': 204,\n", " 'HostId': '0ZOL6wChH/cuxu9indqL91bZFnXLhU/BjOoHv8gUWZkmBgUVqRtHV3V++2YJXMxjTTnCc68br8c=',\n", " 'RequestId': '974C2E2BFFE2C71B'}}" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s3.mv('s3://dev-rsaporta/DataAnalyticsDept/clf.pkl',\n", " 's3://dev-rsaporta/DataAnalyticsDept/one_offs/sklearn_clf/clf.pkl')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "top" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.1" } }, "nbformat": 4, "nbformat_minor": 0 }