The week of May 24 we were notified by a small group of customers of incidents in the resolution of accounts in their deployments of the platform. In particular, we were able to establish two cases of failure: the account was not responding (404) or an incorrect account was being charged.
To understand the problem, it must be understood that Modyo operates with a multi-account and multi-domain scheme that depends on an internal dynamic routing system that decides, according to the configured attributes, which account and site should be deployed in each request, regardless of whether it is from the frontend or from the administrative console. This routing component also operates with a Redis-based caching system that prevents the same account and site from being loaded from the database in subsequent requests.
After thoroughly investigating the two established failure scenarios, a possible thread safety problem was detected in the account resolution component that made use of instance variables in a place in the framework where they were not evaluated in isolation. The behavior was replicated and a security patch was generated that should prevent the problem from recurring. The problem was present between versions 9.1.4 and 9.1.20 of the platform, affecting only enterprise customers with more than one account configured in their deployments.
The following fixes were applied in version 9.1.21:
- Remove caching in Redis on account resolution, to avoid persistence of any eventual bad resolution.
- Remove the use of instance variables in the Rack resolution middleware (thread safety issue).
Problems related to threads are difficult to detect and replicate, and in many cases they will tend to occur very occasionally, affecting only in very particular situations. Nevertheless, we strongly recommend upgrading to the latest version of the platform in order to prevent the problem from occurring again.
We are sorry for any inconvenience the problem may have caused. To date we have only been notified of a very small number of cases.
Jose Antonio Silva