Tap Shopify Performance Enhancements

.gitignore

@@ -102,3 +102,4 @@ rsa-key
 tags
 singer-check-tap-data
 state.json
+publish_to_artifactory.sh

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 1.2.0
+  * Updated Tap to get orders asynchronously, using asyncio and aiohttp libraries. To use Async method can be configured in the tap-config.
+
+## 1.1.9
+  * Updated Tap to allow for Private Shopify App which can be configured in the tap-config.
+
 ## 1.1.8
   * Uses patternProperties to match extra fields on transactions receipts [#33](https://github.com/singer-io/tap-shopify/pull/33)
 

README.md

@@ -1,5 +1,10 @@
 # tap-shopify
 
+## Source Repository
+* Git Repo: https://github.com/singer-io/tap-bing-ads
+* To Sync Fork: https://help.github.com/articles/syncing-a-fork/
+
+## Information
 This is a [Singer](https://singer.io) tap that produces JSON-formatted data
 following the [Singer
 spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
@@ -21,7 +26,6 @@ This tap:
 - When Metafields are selected, this tap will sync the Shopify store's top-level Metafields and any additional Metafields for selected tables that also have them (ie: Orders, Products, Customers)
 
 ## Quick Start
-
 1. Install
 
     pip install tap-shopify
@@ -32,9 +36,13 @@ This tap:
 
    ```json
     {
-        "start_date": "2010-01-01",
-        "api_key": "<Shopify API Key>",
-        "shop": "test_shop"
+        "start_date": "2019-01-01T00:00:00Z",
+        "end_date": "2019-01-31T00:00:00Z",
+        "shop": "test_shop",
+        "is_private_app": true,
+        "api_key": "<<Shopify API Key>>",
+        "api_password": "<<Shopify API Password (if private app)>>",
+        "use_async": true
     }
     ```
 
@@ -56,6 +64,32 @@ This tap:
 
     tap-shopify -c config.json --catalog catalog-file.json
 
+## Performance
+
+### Shopify Constraints
+Shopify API's throttle is designed to allow your app to make unlimited requests at a steady rate over time while also having the capacity to make infrequent bursts. The throttle operates using a [leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket) algorithm. The bucket size and leak rate properties determine the API's burst behavior and call rate.
+
+|                 | Bucket Size | Leak Rate  | Max. Results/Call  |
+| --------------- |------------:| ----------:| ------------------:|
+| Shopify Regular | 40          | 2/second   | 250                |
+| Shopify Plus    | 80          | 4/second   | 250                |
+
+### Async Logic
+1. I found that using Shopify's SDK is slower by almost a factor of 10 compared to calling the REST endpoints.
+    * So, we scrap that, and just make calls directly to the REST endpoints.
+2. Check if Shop is Regular or Plus to determine Bucket Size.
+3. Check the total number of orders for the date range you are pulling data for.
+4. Num. Pages = Total Number of Orders/250
+5. Each call you can get one page and you are allowed to make 40 calls (or 80 calls).
+    *  Num. orders you can retrieve by making all 40 calls (or 80 calls) asynchronously = 10,000 (or 20,000)
+
+6. Say, total number of orders = 100,000.
+    * Then, Num. Pages = 100,000/250 = 400
+    * We cannot make 400 calls asynchronously, so we chunk them into 40 calls each, which gives us a list A that contains nested lists of 40 calls.
+    * We iterate through list A, and using `asyncio` and `aiohttp` we make 40 asynchronous calls, retrieve the results. Then, move onto the next 40 calls.
+    * Then, finally return all the results by page order.
+
+
 ---
 
 Copyright &copy; 2019 Stitch

install_local.sh

@@ -0,0 +1,42 @@
+#! /bin/bash
+get_value_from_file() {
+    key=$1
+    file=$2
+
+    val=( $(cat $file | grep -Eo "${key}=.{1,50}" | sed "s/^${key}=\"\(.*\)\",$/\1/") )
+    echo $val
+}
+
+
+
+GREEN='\033[0;32m'
+NC='\033[0m' # No Color
+YELLOW='\033[0;33m'
+RED='\033[0;31m'
+
+
+
+lib_version=( $(get_value_from_file 'version' setup.py) )
+lib_name=( $(get_value_from_file 'name' setup.py) )
+
+
+
+python3 setup.py sdist bdist_wheel
+pip uninstall --y $lib_name
+if pip install ./dist/${lib_name}-${lib_version}.tar.gz ; then
+    rm -rf ./build
+    rm -rf ./dist
+    rm -rf ./*.egg-info
+    echo 
+    echo -e "${GREEN}##########################################################${NC}"
+    echo -e "${GREEN}  Successfully Installed: ${YELLOW}${lib_name}-${lib_version}${NC}"
+    echo -e "${GREEN}##########################################################${NC}"
+    echo 
+else
+    echo 
+    echo -e "${RED}##########################################################${NC}"
+    echo -e "${RED}  Failed to Install: ${YELLOW}${lib_name}-${lib_version}${NC}"
+    echo -e "${RED}##########################################################${NC}"
+    echo 
+fi
+

sample_config.json

@@ -1,4 +1,9 @@
 {
+  "start_date": "2019-01-01T00:00:00Z",
+  "end_date": "2019-01-31T00:00:00Z",
+  "shop": "someshop",
+  "is_private_app": true,
   "api_key": "foobarcharnockbat",
-  "start_date": "2017-01-01T00:00:00Z"
+  "api_password": "batnockcharbarfoo",
+  "use_async": true
 }

setup.py

@@ -3,7 +3,7 @@
 
 setup(
     name="tap-shopify",
-    version="1.1.8",
+    version="1.2.0",
     description="Singer.io tap for extracting Shopify data",
     author="Stitch",
     url="http://github.com/singer-io/tap-shopify",
@@ -18,6 +18,8 @@
             'pylint',
             'ipdb',
             'requests==2.20.0',
+            'aiohttp==3.5.4',
+            'asyncio==3.4.3'
         ]
     },
     entry_points="""