Subscribe
06 Nov 2023, 05:45 PM
Update on the issue with display of orders and positions on November 6, 2023
At the market open today, about 20% of our clients on a silo experienced issues with checking the latest status of their orders and positions while order placement continued unaffected. This issue was limited to one of the many independent “silos” which are designed to operate independently, which is why 80% of the clients were unaffected.
Timeline:
09:15 AM: Order and position status updates were affected, but order placement was unaffected.
10:00 AM: The processes that handle streaming order and position updates from our order management system (OMS) slowed down inexplicably. These processes have failovers that mirror them 1:1, but that mirrors also exhibited the same behaviour. We were unable to ascertain the root cause at this point. We deployed an ad-hoc solution to re-sync the slowed down order and position stream data, but the same behaviour caused this to slow down significantly.
10:38 AM: To bring the slowed streams under control, we stopped order placement for the affected set of users on the silo.
11:22 AM: The orders and positions were synced on Kite, making the updated orders and positions data available to clients.
11:38 AM: After ensuring the stability of the streams, we resumed allowing clients to exit their positions and orders but fresh orders were still blocked.
12:09 PM: All restrictions on order placement are removed.
We have put in additional checks to ensure that the issue doesn’t arise again. We are still investigating the root cause and will share a root cause analysis (RCA) of the issue soon.
We sincerely apologize for the issues caused today. We are still investigating the root cause. If you were affected, please create a ticket and someone from our team will reach out to you. We have waived off call and trade and DP charges for the day.