15 Jun 2022, 06:55 PM

Issue with the display of orders, holdings & positions (NSE, BSE, MCX)

Duration:  04:21 PM to 04:59 PM

Incident: Internet Connectivity to our trading platform (along with many other websites) was impacted from 04:21 PM to 04:59 PM owing to intermittent connectivity issues via the Cloudflare network for users on certain ISPs. This caused intermittent issues w.r.t. the display of orders, holdings & positions temporarily on Kite for some of our users. The users were able to reach our application post multiple retries even when the issue was ongoing though. For those users who were impacted Call and trade desk was open.

Cause:

  1. The internet connectivity to our trading platform was impacted due to a problem at Cloudflare’s end (Cloudflare is a vendor that provides CDN, network capacity, and DDoS protection to Zerodha’s public-facing internet properties). 

  2. A bug in their internal WAF-managed ruleset deployment process resulted in old ruleset being referenced when not needed. Specifically, several identical replicas of the several Cloudflare Managed Rulesets existed in their systems. 

  3. Purging of unnecessary versions to perform database cleanup resulted in increased error logging in their internal systems which triggered the incident.


Approx. clients affected: 7000 - 9000

Update: 14-07-2022 Root cause analysis (RCA) - RCA glitch dated 15-06-2022 - NSE