Skip to content

Instantly share code, notes, and snippets.

@wowkin2

wowkin2/Readme.md

Last active September 25, 2024 19:10
Show Gist options
  • Save wowkin2/079844c867a1a06ce15ea1e4ffdee87c to your computer and use it in GitHub Desktop.
Save wowkin2/079844c867a1a06ce15ea1e4ffdee87c to your computer and use it in GitHub Desktop.
Download ZIP
Solution: "Exceeded 4 calls per second for api client" Python Shopify API - shopify_python_api
Raw
Readme.md

Solution for API call limit "shopify_python_api"

If you are using Python Shopify API and getting following error
"Exceeded 4 calls per second for api client. Reduce request rates to resume uninterrupted service."
but want your script to continue working with some timeout after that,
you can use following script from shopify_limits_patch.py.

For that just copy shopify_limits_patch.py to your project and import shopify_limits_patch.

Or if you want to call it implicitly import it, remove last line patch_shopify_with_limits()
and call it before all your shopify calls.

This is solution for issue described here: Shopify/shopify_python_api#256

If you have any suggestions for code - leave comment :)

Raw
shopify_limits_patch.py
import time
import pyactiveresource.connection
from shopify.base import ShopifyConnection
def patch_shopify_with_limits():
func = ShopifyConnection._open
def patched_open(self, *args, **kwargs):
while True:
try:
return func(self, *args, **kwargs)
except pyactiveresource.connection.ClientError as e:
if e.response.code == 429:
retry_after = float(e.response.headers.get('Retry-After', 4))
print('Service exceeds Shopify API call limit, '
'will retry to send request in %s seconds' % retry_after)
time.sleep(retry_after)
else:
raise e
ShopifyConnection._open = patched_open
patch_shopify_with_limits()
Raw
validation.py
import os
from threading import Thread
import shopify
import shopify_limits_patch
def foo():
for _ in range(15):
shopify.Variant.find()
def main():
# Add your Shopify URL here: "https://<shop_name>.myshopify.com/admin"
shopify.ShopifyResource.set_site(os.environ.get('SHOPIFY_URL'))
for i in range(15):
t = Thread(target=foo)
t.start()
if __name__ == '__main__':
main()
@wowkin2
Copy link
Author

wowkin2 commented Jul 24, 2018

Official Shopify documenation API call limit:
https://help.shopify.com/en/api/getting-started/api-call-limit

@coler-j
Copy link

coler-j commented Feb 26, 2019

Do you have any guidance on integrating this patch into a Django application?

@sameerkumar18
Copy link

sameerkumar18 commented Mar 24, 2019 •
edited
Loading

Thanks a ton for this @wowkin2
I was also able to handle Shopify server 5XX errors with this approach.

@coler-j I used this with django, just try not to import it inside any view. Preferably call this function after importing shopify.
Example code snippet -

import shopify
import pyactiveresource.connection
from shopify.base import ShopifyConnection


# For handling Shopify 429 - https://gist.github.com/wowkin2/079844c867a1a06ce15ea1e4ffdee87c
def patch_shopify_with_limits():
    func = ShopifyConnection._open

    def patched_open(self, *args, **kwargs):
        count = 0
        while True:
            count += 1
            try:
                return func(self, *args, **kwargs)

            except pyactiveresource.connection.ClientError as e:
                if e.response.code == 429:
                    # Checks for Retry-After header in 429 else defaults to 4 sec
                    retry_after = float(e.response.headers.get('Retry-After', 4))
                    log.info(f'Service exceeds Shopify API call limit, '
                             f'will retry to send request in {retry_after} seconds')
                    time.sleep(retry_after)
                else:
                    raise e
    ShopifyConnection._open = patched_open


patch_shopify_with_limits()

@api_view(http_method_names=['post', ])

@sillycube
Copy link

sillycube commented Mar 24, 2020 •
edited
Loading

I tried to implicitly import it, removed last line patch_shopify_with_limits() and called it before all my shopify calls. But I got errors for several endpoints. Seems like the recursion was using all memory. My django app will die after running for 1 - 2 days. After that I gracefully kill gunicorn and the app will be up again. Currently, I am trying to replace the while loop with a for loop running for several times. Just not sure if it can fix the issue yet.

Internal Server Error: /app_spo/getOptimizedDetail

RecursionError at /app_spo/getOptimizedDetail
maximum recursion depth exceeded while calling a Python object

It indicates that this line gives the error:
File "/var/www/app.seo-product-optimizer.com/app_spo/views/listProductAjax/shopify_limits_patch.py" in patched_open
13. return func(self, *args, **kwargs)

Request Method: GET
Django Version: 1.11.26
Python Version: 3.7.5

update:
Turn out that I am sending a bunch of ajax requests to my server. When there are too many requests, it will use up the memory and give the error.

@deejax
Copy link

deejax commented Jun 8, 2021

works like a charm, thanks a lot.

@shriDeveloper
Copy link

shriDeveloper commented Jul 4, 2021 •
edited
Loading

Is there a way to do the same thing in requests library. I am using python requests to POST data to shopify (add new image) endpoint.

new_image = json.loads(util.shopify_post(f"https://{shop_url}/admin/api/2021-04/products/{product_id}/images.json" , {
                    "image": {
                        "attachment": optimized_image,
                        "filename":product_format+'.jpg',
                    }
                }).text)

where, the above code is getting called inside a for loop.

Once i hit the shopify benchmark for requests (40 per bucket) . it throws a error saying "Exceeded 2 calls per second for api client" and everything crashes.

Is there anything that can be done over here ?

i tried by putting sleep(3) seconds for every image being create inside the loop. it works . but don't think this is scalable enough and fast.

Any help would be highly appreaciated.

@wowkin2
Copy link
Author

wowkin2 commented Jul 4, 2021 •
edited
Loading

@shriDeveloper check for status_code and do retry or use some python queue. And that's it :)

@shriDeveloper
Copy link

shriDeveloper commented Jul 5, 2021 •
edited
Loading

@wowkin2 thanks for advice. i did some try with ratelimit library but didn't help too. retry seems good for now.

if reponse.status_code == 429:
       time.sleep(3)
       #shopify call again .

@bekhzod91
Copy link

I tried to implicitly import it, removed last line patch_shopify_with_limits() and called it before all my shopify calls. But I got errors for several endpoints. Seems like the recursion was using all memory. My django app will die after running for 1 - 2 days. After that I gracefully kill gunicorn and the app will be up again. Currently, I am trying to replace the while loop with a for loop running for several times. Just not sure if it can fix the issue yet.

Internal Server Error: /app_spo/getOptimizedDetail

RecursionError at /app_spo/getOptimizedDetail maximum recursion depth exceeded while calling a Python object

It indicates that this line gives the error: File "/var/www/app.seo-product-optimizer.com/app_spo/views/listProductAjax/shopify_limits_patch.py" in patched_open 13. return func(self, *args, **kwargs)

Request Method: GET Django Version: 1.11.26 Python Version: 3.7.5

update: Turn out that I am sending a bunch of ajax requests to my server. When there are too many requests, it will use up the memory and give the error.

When you call patch_shopify_with_limits much times you get this errors, try replace to this code


import pyactiveresource.connection
from shopify.base import ShopifyConnection


_shopify_open = ShopifyConnection._open


def patch_shopify_with_limits():
    def patched_open(self, *args, **kwargs):
        while True:
            try:
                return _shopify_open(self, *args, **kwargs)
            except pyactiveresource.connection.ClientError as e:
                if e.response.code == 429:
                    retry_after = float(e.response.headers.get('Retry-After', 4))
                    print('Service exceeds Shopify API call limit, '
                          'will retry to send request in %s seconds' % retry_after)
                    time.sleep(retry_after)
                else:
                    raise e

    ShopifyConnection._open = patched_open


patch_shopify_with_limits()

@theplen
Copy link

theplen commented Dec 16, 2022

Brilliant! Thanks!

@macksal
Copy link

macksal commented Jun 20, 2024

Still working in 2024!

@b4p3p
Copy link

b4p3p commented Sep 25, 2024

It works perfectly in late 2024, the only change i made is

retry_after = float(e.response.headers.get('retry-after', 4))

I created a shopify_patched module, and inside the init file, i added

from .shopify_limits_patch import shopify

To import the library, i replaced all:

import shopify

with

from app_api.api_modules.shopify_patched import shopify

Is that correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment