Many events on the site have repercussion beyond the direct object where they acted up on and executing this repercussion can take some time. If we waited for all this to happen before showing the next page to the user, the site would be extremely slow.
These are some examples of such activities:
- Measuring a bra. When you enter measurements for a bra, not only that one bra needs to have its numbers updated on the database, but also need to recalculate the averages with all the other bras the same size, and then recalculate the interpolations to estimate surrounding sizes measurements.
- Adding a picture. Every picture needs processing done before displaying. That is to strip EXIF information and to generate the thumbnails that the site will be using so next time they are requested they are immediately available and don't take the time to generate the thumbnail on demand, this is a problem because the original files are stored on a cloud service which makes it slow to generate them.
- Send notifications about listings. When a new listing gets added we need to check which users are subscribed to such characteristics and let them know about it.
- And so on...
In order to make the site respond faster, we delay many of these operations to a separate process, an asynchronous process that receives all these tasks and process them later in time.
Up until last night, we had one of these workers executing all the tasks one after the other. Sadly it would eventually choke when trying to generate many thumbnails for many big images simultaneously as the server would run out of memory and stop the worker. This caused all the following queued tasks not to execute, and up on restarting some of them would get lost.
What we did last night was to create a new separate process to handle only this potentially problematic tasks, and leave the other only for simple tasks. This way we are ensuring that no other tasks get lost and if any image gets choked, it is not a loss as the next time the thumbnails are requested the task will be executed.