How to optimize and reduce the load on your Alumio environment
This lesson will provide insights into various ways to optimize your Alumio environment. This may be needed to reduce the load on your environment and guarantee the stability of the various integrations, especially during a surge/spike in traffic.
What are traffic spikes, and how to deal with them?
Traffic spikes usually occur when the scheduler is not optimally configured, causing many processes to run simultaneously. This can also happen if a subscriber receives more data than usual, leading to increased execution times that overlap with other processes, which should be avoided.
Optimizing the scheduler is essential to mitigate this. Additionally, it's important to manage the number of tasks executed by each route. We recommend avoiding the execution of hundreds of tasks per minute for each route. Prioritizing the scheduler tasks and reducing the amount of data captured during incoming processes can also help.
See a complete breakdown of how to optimize your scheduling jobs here: https://forum.alumio.com/t/optimizing-your-scheduled-jobs/204
How can you reduce the level of detail in log files?
Alumio gives users the possibility to enable logging in various stages of the process. By default, Alumio will log all occurrences within integrations. This cannot be disabled and will always be logged in full detail. Examples of these logs are:
- Filtered entities
- Created tasks
- Alumio process-related information
On top of these logs, Alumio gives users the ability to enable logging for the systems they integrate. Think of authentication logs, API logs, database logs, webhook logs, etc. These logs are very useful for debugging any issues related to the systems integrated. However, in some cases, these logs can cause lots of load on the Alumio environment. Examples of when this might happen are:
- A lot of logs are being created. For instance, Alumio pulls 100k products and does an API call for each product to retrieve the stock and price from another system
- The log files are extremely big
To prevent this from causing issues, it’s advised to stick to the Simple and Long logging types and avoid Full logging. While Full seems very useful, it is advised to only use this during the testing phase. As the name already suggests, Full, will store the complete requests and responses. Simple will only store the header information while Long will store up to 10,000 characters, which should be sufficient to debug any issues that may arise.
How can you reduce the number of tasks exported per execution?
Most integrations built using Alumio are asynchronous. This means that the data is being synchronized in the background and that there are two steps to each integration: filling the queue and processing the queue.
In some cases, you may opt to go with a real-time processing integration, essentially reducing the process to a single step: filling the queue and immediately processing it. Depending on the number of tasks processed, this may cause a high load on your Alumio environment due to the complexity of your integration. Some integrations require additional API calls to complete the data object, or each task might contain a lot of information. Such integrations could lead to high usage of server resources when a high number of tasks are processed within each execution.
It is advised to reduce this number by optimizing your scheduled jobs, reducing the number of entities sent per webhook, or introducing batch processing. Batch data integration entails the consolidation and processing of data in predefined batches at scheduled intervals. This approach, which Alumio enables, accumulates data over a period, often hours or days, before transferring it en masse to the target system for processing.
How can you reduce the number of scheduled jobs that run in parallel?
An Alumio environment maybe configured to start many scheduled jobs in parallel. This could lead to a high load on your Alumio environment. For this reason, it is advised to spread the number of scheduled jobs to run across the hour.
*Once again, it is advised that you read the information on optimizing your scheduled jobs in our Forum: https://forum.alumio.com/t/optimizing-your-scheduled-jobs/204
Upgrading your Alumio environment
Following the tips given above should help you keep your Alumio environment within acceptable load levels in most cases. When this is not the case, it is best to consider upgrading your Alumio environment. You can do this by reaching out to our Support Team.
