I have a process for updating info in my database from an API call. The API call usually happens from a page for the user one at a time. Now I built a snippet that pulls the list (100 or so) db entries that need updating and I loop through calling my REST api and then updating the DB with the results. But, it gets a 504 Gateway error after 10 or so calls (not to mention locks my site for user traffic). I’m on MODX Cloud.
What’s a better way to do this kind of task?
Paul
If you built the REST api yourself, maybe you can add a batch update endpoint. Send just one request with the appropriate IDs to update; save yourself (and the server) the overhead of 99 requests/responses.
If the process can happen asynchronously, you can also schedule tasks into a message queue. Something like Scheduler. Or a simple cron job running every X minutes to process what needs to be processed.
If one user change can lead to hundred(s?) of required database changes, then it may also be worth making sure you data is modelled correctly. Perhaps an extra layer of abstraction can reduce the need?
Thanks @markh All your suggestions are good. In this case, the Scheduler (some sort of embedded task) that doesn’t take nginx resources much (maybe with a delay between calls) is nice.
I do not own the API, but will contact the vendor and see about a batch. That would absolutely solve it.
How would I set up a cron job? I’m a Windows C# programmer by trade, so a lot of resources programmers available in this environment are foreign to me. But anything I can add to MODX Cloud, or anything with good sample code I can usually figure out.
MODX Cloud has a dedicated documentation page about using cron on their platform.