{"id":365846,"date":"2023-12-05T18:03:47","date_gmt":"2023-12-05T12:33:47","guid":{"rendered":"https:\/\/zerodha.com\/z-connect\/?p=365846"},"modified":"2024-02-02T12:37:39","modified_gmt":"2024-02-02T07:07:39","slug":"an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months","status":"publish","type":"post","link":"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months","title":{"rendered":"An apology for the two consecutive tech issues in the last two months"},"content":{"rendered":"<p>In the <a href=\"https:\/\/zerodha.com\/z-connect\/zerodha\/why-zerodha\/business-updates-13-years-of-zerodha-and-why-we-do-what-we-do\">business updates post<\/a> I shared in August this year, I mentioned how we hadn\u2019t had any large tech issues for a couple of years. Unfortunately, we have had two episodes in quick succession in the last two months, affecting between 5 and 20% of our active customers.<\/p>\n<p>As a broker, we have multiple external dependencies. To name a few:<\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Exchanges and depositories<\/li>\n<li>Data centres: physical &amp; cloud<\/li>\n<li>Leased lines for connectivity between exchanges and data centres<\/li>\n<li>Our Execution Management System (EMS) vendor<\/li>\n<li>Cloudflare for SSL<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>The issues on Nov 6th and Dec 4th were triggered due to edge cases with our external dependencies. This is no excuse, and I understand that, as a platform, we are responsible for all the issues you face. But I wanted to share with you what went wrong and what we are doing about it.<\/p>\n<p>The Nov 6th issue was due to an unscheduled update in the anti-malware monitoring service from our EMS vendor, which started throttling our servers. You can check the detailed RCA <a href=\"https:\/\/zerodha.com\/marketintel\/bulletin\/363900\/update-on-the-issue-with-display-of-orders-and-positions-on-november-6-2023\">here<\/a>.<\/p>\n<p>Yesterday\u2019s issue seems to be because of an exponentially larger number of customer password reset requests that caused login issues. On Monday morning, the system that notifies users of logins from new geographical locations based on IP addresses sent out an unexpectedly large number of alerts. We discovered that this was the result of an increase in the geo-location accuracy of the IP\/geo-location database that we use. A routine update of this database happened over the weekend. We believe this led to a large influx of password reset requests from confused users, putting a strain on our login systems and resulting in login failures. We will share a detailed RCA on the disclosure page soon.<\/p>\n<p>We now have put in place fixes to ensure these types of cases don\u2019t affect our platform in the future. While we continuously put a lot of effort into ensuring all types of scenarios are factored in proactively, it is impossible for any technology platform to cover all edge cases. A large number of users attempting to reset their passwords due to a notification triggered by an increase in IP geolocation accuracy was one of those scenarios. Please be assured that, as a team, we are doing whatever we can to ensure the platform&#8217;s stability.<\/p>\n<p>Social media has been a double-edged sword for us. We have significantly benefited from it since it made interacting with our customers easier and helped spread the word about our business. But the flip side is that we get a disproportionate number of tweets on a day like yesterday compared to the issue. On a normal day, we receive about 3500 to 4000 customer tickets, and about 10 lakh users trade daily. Yesterday, we had about 7700 tickets, an increase of about 4000. We believe that ~10% of active users faced disruptions intermittently for about 30 minutes. I am in no way saying this to minimise the issues traders faced, but just to highlight that we have a younger and more vocal online audience. Given that the markets moved significantly, there was a lot of interest from non-transacting users logging in to check their portfolios, making the disruption seem much larger than otherwise.<\/p>\n<p>Once again, I want to assure you that the stability and reliability of the platform are our top priorities. We are working hard towards that in whatever way possible. We are extremely sorry for the inconvenience to those who were affected. If there have been any losses due to the incidents, create a ticket, and our team will try to get back to you as soon as possible with the best way to resolve them.<\/p>\n<p>Sorry again,<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In the business updates post I shared in August this year, I mentioned how we hadn\u2019t had any large tech issues for a couple of years. Unfortunately, we have had two episodes in quick succession in the last two months, affecting between 5 and 20% of our active customers. As a broker, we have multiple [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[531],"tags":[],"class_list":["post-365846","post","type-post","status-publish","format-standard","hentry","category-general"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v23.5 (Yoast SEO v26.6) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>An apology for the two consecutive tech issues in the last two months &#8211; Z-Connect by Zerodha<\/title>\n<meta name=\"description\" content=\"In the business updates post I shared in August this year, I mentioned how we hadn\u2019t had any large tech issues for a couple of years. Unfortunately, we\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"An apology for the two consecutive tech issues in the last two months\" \/>\n<meta property=\"og:description\" content=\"In the business updates post I shared in August this year, I mentioned how we hadn\u2019t had any large tech issues for a couple of years. Unfortunately, we\" \/>\n<meta property=\"og:url\" content=\"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months\" \/>\n<meta property=\"og:site_name\" content=\"Z-Connect by Zerodha\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/zerodha.social\" \/>\n<meta property=\"article:published_time\" content=\"2023-12-05T12:33:47+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-02-02T07:07:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/zerodha.com\/z-connect\/wp-content\/uploads\/2024\/02\/thumb.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"675\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Nithin Kamath\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@nithin0dha\" \/>\n<meta name=\"twitter:site\" content=\"@zerodhaonline\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nithin Kamath\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months\",\"url\":\"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months\",\"name\":\"An apology for the two consecutive tech issues in the last two months &#8211; Z-Connect by Zerodha\",\"isPartOf\":{\"@id\":\"https:\/\/zerodha.com\/z-connect\/#website\"},\"datePublished\":\"2023-12-05T12:33:47+00:00\",\"dateModified\":\"2024-02-02T07:07:39+00:00\",\"author\":{\"@id\":\"https:\/\/zerodha.com\/z-connect\/#\/schema\/person\/03de5b6b47eb838a0067a9d11a1ffaf6\"},\"description\":\"In the business updates post I shared in August this year, I mentioned how we hadn\u2019t had any large tech issues for a couple of years. Unfortunately, we\",\"breadcrumb\":{\"@id\":\"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/zerodha.com\/z-connect\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"General\",\"item\":\"https:\/\/zerodha.com\/z-connect\/category\/general\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"An apology for the two consecutive tech issues in the last two months\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/zerodha.com\/z-connect\/#website\",\"url\":\"https:\/\/zerodha.com\/z-connect\/\",\"name\":\"Zerodha product and business updates\",\"description\":\"Z-Connect is Zerodha&#039;s official blog. Get the latest updates, product announcements news, and insights all in one place.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/zerodha.com\/z-connect\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/zerodha.com\/z-connect\/#\/schema\/person\/03de5b6b47eb838a0067a9d11a1ffaf6\",\"name\":\"Nithin Kamath\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/zerodha.com\/z-connect\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/zerodha.com\/z-connect\/wp-content\/uploads\/2019\/05\/unnamed-1-96x96.jpg\",\"contentUrl\":\"https:\/\/zerodha.com\/z-connect\/wp-content\/uploads\/2019\/05\/unnamed-1-96x96.jpg\",\"caption\":\"Nithin Kamath\"},\"description\":\"Founder &amp; CEO @ Zerodha\",\"sameAs\":[\"https:\/\/nithinkamath.me\",\"https:\/\/x.com\/nithin0dha\"],\"url\":\"https:\/\/zerodha.com\/z-connect\/author\/nithin-kamath\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"An apology for the two consecutive tech issues in the last two months &#8211; Z-Connect by Zerodha","description":"In the business updates post I shared in August this year, I mentioned how we hadn\u2019t had any large tech issues for a couple of years. Unfortunately, we","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months","og_locale":"en_US","og_type":"article","og_title":"An apology for the two consecutive tech issues in the last two months","og_description":"In the business updates post I shared in August this year, I mentioned how we hadn\u2019t had any large tech issues for a couple of years. Unfortunately, we","og_url":"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months","og_site_name":"Z-Connect by Zerodha","article_publisher":"https:\/\/www.facebook.com\/zerodha.social","article_published_time":"2023-12-05T12:33:47+00:00","article_modified_time":"2024-02-02T07:07:39+00:00","og_image":[{"width":1200,"height":675,"url":"https:\/\/zerodha.com\/z-connect\/wp-content\/uploads\/2024\/02\/thumb.png","type":"image\/png"}],"author":"Nithin Kamath","twitter_card":"summary_large_image","twitter_creator":"@nithin0dha","twitter_site":"@zerodhaonline","twitter_misc":{"Written by":"Nithin Kamath","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months","url":"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months","name":"An apology for the two consecutive tech issues in the last two months &#8211; Z-Connect by Zerodha","isPartOf":{"@id":"https:\/\/zerodha.com\/z-connect\/#website"},"datePublished":"2023-12-05T12:33:47+00:00","dateModified":"2024-02-02T07:07:39+00:00","author":{"@id":"https:\/\/zerodha.com\/z-connect\/#\/schema\/person\/03de5b6b47eb838a0067a9d11a1ffaf6"},"description":"In the business updates post I shared in August this year, I mentioned how we hadn\u2019t had any large tech issues for a couple of years. Unfortunately, we","breadcrumb":{"@id":"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/zerodha.com\/z-connect\/general\/an-apology-for-the-two-consecutive-tech-issues-in-the-last-two-months#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/zerodha.com\/z-connect\/"},{"@type":"ListItem","position":2,"name":"General","item":"https:\/\/zerodha.com\/z-connect\/category\/general"},{"@type":"ListItem","position":3,"name":"An apology for the two consecutive tech issues in the last two months"}]},{"@type":"WebSite","@id":"https:\/\/zerodha.com\/z-connect\/#website","url":"https:\/\/zerodha.com\/z-connect\/","name":"Zerodha product and business updates","description":"Z-Connect is Zerodha&#039;s official blog. Get the latest updates, product announcements news, and insights all in one place.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/zerodha.com\/z-connect\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/zerodha.com\/z-connect\/#\/schema\/person\/03de5b6b47eb838a0067a9d11a1ffaf6","name":"Nithin Kamath","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/zerodha.com\/z-connect\/#\/schema\/person\/image\/","url":"https:\/\/zerodha.com\/z-connect\/wp-content\/uploads\/2019\/05\/unnamed-1-96x96.jpg","contentUrl":"https:\/\/zerodha.com\/z-connect\/wp-content\/uploads\/2019\/05\/unnamed-1-96x96.jpg","caption":"Nithin Kamath"},"description":"Founder &amp; CEO @ Zerodha","sameAs":["https:\/\/nithinkamath.me","https:\/\/x.com\/nithin0dha"],"url":"https:\/\/zerodha.com\/z-connect\/author\/nithin-kamath"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/posts\/365846","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/comments?post=365846"}],"version-history":[{"count":4,"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/posts\/365846\/revisions"}],"predecessor-version":[{"id":365850,"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/posts\/365846\/revisions\/365850"}],"wp:attachment":[{"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/media?parent=365846"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/categories?post=365846"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zerodha.com\/z-connect\/wp-json\/wp\/v2\/tags?post=365846"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}