We were not down today
Dear traders,
If you happened to notice the outcry on social media (we were trending on Twitter India) and breaking news reports by some major media houses saying our systems were down today, but were surprised because you were able to trade smoothly, we are in the same boat as you. We had the highest number of concurrent users today and processed the largest number of orders ever. There was absolutely no downtime except for two extremely small hiccups that affected a very small portion of our users:
- Between 9:15 to 9:18, we had intermittent leased line re-connections with NSE at our data centre B. This new data centre went live last week, and the orders from a small segment of users pass through the new leased lines there. Orders flowing through this particular line were slow or rejected for the first couple minutes, and it resolved immediately on its own.
- There seems to have been an unrelated, intermittent, network/routing issue between our CDN (Content Delivery Network) CloudFlare and certain internet providers in a few cities (this usually happens with Jio), where the Kite mobile app’s requests to fetch static data that is required during open, failed. Retrying a couple of times, or reconnecting the internet should’ve resolved this. These sort of random network drops are common across services, and every time we spot these issues, we raise tickets with the network service providers. From our logs, we could see that only a handful of users would have faced this.
These were momentary and would have been noticed by only a tiny fraction of the 1+ million users who logged into Kite today. None of our systems were down or overloaded. In fact, they handled peak loads (the highest ever for us) smoothly. However, we were as surprised as you were, by the inexplicable troll campaign on Twitter, and by the media reports that claimed that we were down with zero fact-checking.
Traffic graph today from 9:15 to 3:30. Peak activity, as always, is at 9:15 AM. There are no sharp or abnormal drops. Counts on Y-axis are hidden.
Traffic graph for Kite from 9:15 to 3:30
Graph of orders from 9:15 to 3:30 for today
Order rates from 9:15 to 3:30 PM (count on Y-axis is hidden). Peak orders at 9:15 as always, and an even rate throughout the rest of the day with no gaps or drops. We processed close to 8.5 million orders today.
Data centre B
As we have discussed multiple times before, we use Refinitiv (formerly Thomson Reuters), like most of the industry, to power our exchange connectivity and order management systems (OMS). It took us close to two years to commission a second setup of this system including acquiring new racks and servers at a different data centre, multiple leased lines from all exchanges (where each line can take weeks to months to commission). It went live silently last week. What this means is that underneath Kite, there are essentially two OMS, technically two Zerodhas, with equal capacity sharing load, and also acting as backups for each other in case of calamities like earthquakes or floods. This also means that it doubles the order handling capacity of the Refinitiv OMS, which has been a source of most technical issues in the past.
The 3-minute leased line connectivity drop at market opening happened at this new data centre today, where only a small portion of our users have been moved internally. You can imagine our surprise when this tiny drop at an isolated data centre for only a small fraction of the users got blown out of proportion, with social media and the news making it look like the whole brokerage was down.
The daily activity and traffic on Kite is several times more than the next biggest broker — over one million active users every day generating over a billion requests. As always, we continue to build and improve our systems and launch new products and features. The silent commissioning of data centre B is one of the many things we have been working on. We also recently pushed the intraday square-off timings of F&O positions to 3:25 PM, giving traders an extra 5 minutes, which the rest of the industry does not. This has been possible thanks to the increased capacity of our setup. More to come soon.
Happy trading,
