tag:status.kornitx.net,2005:/historyKornitX Status - Incident History2024-03-28T15:47:33ZKornitXtag:status.kornitx.net,2005:Incident/198661982024-04-15T05:00:00Z2024-02-16T15:12:27ZKornitX System Maintenance<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>15</var>, <var data-var='time'>05:00</var> - <var data-var='time'>06:00</var> UTC</strong></p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>15:12</var> UTC</small><br><strong>Update</strong> - Please note, this scheduled maintenance has been postponed. The updated date will be communicated once confirmed. Apologies for any inconvenience caused.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>12:21</var> UTC</small><br><strong>Scheduled</strong> - Please be aware that the KornitX Platform will be unavailable from 5am to 6am GMT on Tuesday 20th February 2024 as we will be performing scheduled maintenance at this time.<br /><br />During this time, all services will be unavailable with the exception of our SmartLinks and SFTP Service. Customers will still be able to place orders via our SmartLinks except where real-time data is required from our Platform, for example via our External Pricing API (EPA).<br /><br />During the maintenance period, all versions of our portal will display a holding page indicating the Platform is under maintenance. In addition, any requests sent to our APIs will receive an HTTP 503 response indicating the service is unavailable. Our SFTP server will remain operational so CSV/XML files can continue to be sent to the server. These will be processed through our Platform once the maintenance period has been completed and services are back online.<br /><br />Please subscribe to further updates regarding this maintenance via this System Status Dashboard. We will advise via this dashboard when the maintenance is complete and services have been restored.<br /><br />This maintenance is necessary to apply routine security and scalability updates to our platform. We apologise for any inconvenience.<br /><br />For more information, or if you have any questions, please contact the KornitX Support Team via our Support Portal (https://support.kornitx.net/portal).</p>tag:status.kornitx.net,2005:Incident/163162582023-03-06T06:17:59Z2023-03-06T06:17:59ZKornitX System Maintenance<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>06:17</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance is completed and everything is working as expected. We are re-enabling the remaining backgound services and will continue to monitor but any issues from now should be reported via our Support Portal, https://support.kornitx.net/portal/en/home.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>05:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Scheduled</strong> - During this time, all services will be unavailable with the exception of our SmartLinks and SFTP Service. Customers will still be able to place orders via our SmartLinks except where real-time data is required from our Platform, for example via our External Pricing API (EPA).<br /><br />During the maintenance period, all versions of our portal will display a holding page indicating the Platform is under maintenance. In addition, any requests sent to our APIs will receive an HTTP 503 response indicating the service is unavailable. Our SFTP server will remain operational so CSV/XML files can continue to be sent to the server. These will be processed through our Platform once the maintenance period has been completed and services are back online.<br /><br />Please subscribe to further updates regarding this maintenance via this System Status Dashboard. We will advise via this dashboard when the maintenance is complete and services have been restored.<br /><br />This maintenance is necessary as we are migrating our database to the cloud in order to improve the performance and scalability of our platform. We apologise for any inconvenience.<br /><br />For more information, or if you have any questions, please contact the KornitX Support Team via our Support Portal (https://support.kornitx.net/portal).</p>tag:status.kornitx.net,2005:Incident/132217752022-11-16T22:02:26Z2022-11-16T22:02:26ZSystem Outage<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>22:02</var> UTC</small><br><strong>Resolved</strong> - Our hosting provider has confirmed that network connectivity has been restored. They are continuing to monitor the situation but our services have been operational for some time so this incident will be closed. <br /><br />Further details to follow on the root cause of this incident.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>19:54</var> UTC</small><br><strong>Monitoring</strong> - All services are operational but being monitored.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>19:54</var> UTC</small><br><strong>Update</strong> - The issue is stabilising and connectivity to our services has been restored. We are continuing to monitor and will await confirmation from our hosting provider that the incident is resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>19:10</var> UTC</small><br><strong>Identified</strong> - The issue has been identified as an issue within the network of our hosting provider. They are investigating this and working on a resolution. Furhter updates to follow as they become available.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>18:57</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue with connectivity affecting all services. We will provide further updates as soon as possible.</p>tag:status.kornitx.net,2005:Incident/127530822022-11-09T12:19:38Z2022-11-09T18:37:37ZDatabase Replication<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>12:19</var> UTC</small><br><strong>Resolved</strong> - This incident is now resolved. <br /><br />Note, there is a backlog in processing data exports/reports following this issue so please expect some delays in exporting data from the Platform. Capacity on these services has been increased to help clear this backlog as quickly as possible.<br /><br />If you experience any outstanding issues please contact our support team via any tickets you have open or log a new ticket via our Online Portal (https://support.kornitx.net/portal/en/home) and we will investigate accordingly.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>10:57</var> UTC</small><br><strong>Monitoring</strong> - All remaining Kornit X services have now been re-enabled and are operational. We are continuing to monitor for a short period before updating this incident as resolved.<br /><br />If you experience any outstanding issues please contact our support team via any tickets you have open or log a new ticket via our Online Portal (https://support.kornitx.net/portal/en/home) and we will investigate accordingly.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>09:04</var> UTC</small><br><strong>Update</strong> - We have now completed the process of recreating the replica database. We are currently running some checks before enabling the remaining background processes on this database. This will be done in a phased manner whilst monitoring all services.<br /><br />We will provide a further update soon.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>22:33</var> UTC</small><br><strong>Update</strong> - We are continuing to work on this issue. Unfortunately, the process of backing up the master database, copying the backup files to the relevant server and restoring the replica databases is going to take longer than initially estimated. <br /><br />As a result, this issue will be ongoing until tomorrow. We are continuing to enable functions/integrations as required to minimise any impact and ensure ongoing operations across as many services as possible.<br /><br />If you are still experiencing a significant impact on any services, especially those relating to the processing of orders, please contact our support team via our Online Portal (https://support.kornitx.net/portal/en/home) and we will investigate accordingly.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>17:48</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>15:52</var> UTC</small><br><strong>Update</strong> - We are continuing to work on the process to back up our Master database and then restore the data to the replica databases so that all remaining background processing can be re-enabled. We do not have an ETA on this yet.<br /><br />A lot of services have been re-enabled including the CSV Order Import function and a number of Supplier Integrations.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>13:13</var> UTC</small><br><strong>Update</strong> - The Master Db has been restarted successfully and the replication files are being written again. Some primary services/processes have been re-enabled. We are now starting the process of rebuilding our replica databases after which point we can re-enable all remaining background processing tasks. However, this will take a number of hours.<br /><br />We will try to get as many services/background processing tasks operational as possible during this time to minimise the ongoing impact of this incident.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>13:04</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:43</var> UTC</small><br><strong>Update</strong> - Please note, we are restarting the Master database for our Platform in 5 to 10 minutes. This will result in downtime to all services for a short period. Please expect that no Kornit X services will be operational during this time.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:38</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:08</var> UTC</small><br><strong>Update</strong> - We are continuing to work on this incident.<br /><br />Due to the issues affecting the database replication on our Platform, we need to go through a process of rebuilding our replica databases in order to get all services operational again.<br /><br />As this is a lengthy process that could take a number of hours, we will be re-enabling some of the primary processing services and pointing these to our Master database in the short term in order to get these key services operational.<br /><br />We will update once these primary services are operational and provide further details regarding what is operational and what is still impacted by this issue.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:06</var> UTC</small><br><strong>Investigating</strong> - We have an issue affecting the replication between our databases. This is being investigated currently and updates will be provided as soon as possible. <br /><br />To minimise the impact of this issue, we have disabled our background processes which will impact a number of functions on our Kornit X Platform Portal and the integrations running on the Platform.</p>tag:status.kornitx.net,2005:Incident/127252072022-11-05T11:07:29Z2022-11-05T12:41:18ZConnectivity Issues<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>11:07</var> UTC</small><br><strong>Resolved</strong> - Connectivity has now been restored and all systems are now operational.<br /><br />We will provide additional details in due course.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>10:35</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>10:34</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing a network issue that is affecting our services.<br /><br />We are investigating the issue and will provide an update as soon as possible.</p>tag:status.kornitx.net,2005:Incident/92722222022-02-09T12:01:27Z2022-02-09T21:51:03ZSystem Outage<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>12:01</var> UTC</small><br><strong>Resolved</strong> - Full functionality has been restored and we will continue to monitor.<br /><br />Bare in mind that while the system deals with the backlog there may be delays.<br /><br />We will update shortly with details of our findings.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Identified</strong> - The web tier at platform.custom-gateway.net is recovered and order creation is available again.<br /><br />At this stage while we continue investigations we have not enabled background processing.<br /><br />We will provide a further update at 11:30 UTC.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:26</var> UTC</small><br><strong>Update</strong> - The service outage is being investigated by our Principal Technical team and the issue has been escalated to our senior management. All resources are deployed to resolve the problem as quickly as possible.<br /><br />The incident is affecting our primary database but the exact cause is currently unknown we do not have ETA for resolution as yet. We are working to restore service as quickly as possible. <br /><br />We will provide a further update at 11:00 UTC.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>09:06</var> UTC</small><br><strong>Update</strong> - The investigation into the incident is ongoing.<br /><br />We will provide a further update at 10.00 UTC</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>08:37</var> UTC</small><br><strong>Update</strong> - The investigation into the incident is ongoing.<br /><br />We will provide a further update at 09.00 UTC</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>07:54</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an outage of our platform.<br /><br />An investigation with our hosting partners is underway and an update will be posted again at 08.30 UTC<br />Platform OMS / CPP / APIs /Smartlinks</p>tag:status.kornitx.net,2005:Incident/85224142021-11-14T19:33:30Z2022-02-04T11:40:13ZPlatform Outage<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>19:33</var> UTC</small><br><strong>Resolved</strong> - Services have been restored as of 19:20 UTC.<br /><br />We are still investigating the root cause and will provide more details as soon as possible.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>18:55</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an outage of our web tier.<br /><br />An investigation with our hosting partners is underway and an update will be posted again at 19:30 UTC</p>tag:status.kornitx.net,2005:Incident/74510632021-07-09T07:03:34Z2022-02-04T11:40:14ZConnectivity Issues<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>07:03</var> UTC</small><br><strong>Resolved</strong> - Connectivity has now been restored however we are still working with our hosting providers to investigate the root cause.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>06:23</var> UTC</small><br><strong>Investigating</strong> - We are currently working with our hosting provider to investigate connectivity issues with the platform.<br /><br />Further updates to follow.</p>tag:status.kornitx.net,2005:Incident/53134312020-10-13T10:29:30Z2022-02-04T11:40:14ZNorth America Connectivity Issues<p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>10:29</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:29</var> UTC</small><br><strong>Update</strong> - Our monitoring now indicates that US connectivity has been restored.<br /><br />Many other services appear to have been affected by this issue (as shown by https://downdetector.co.uk and https://livemap.pingdom.com) however unfortunately we do not have any additional information at this moment in time.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>Investigating</strong> - We are aware of connectivity issues with our platform from the USA (specifically the west coast).<br /><br />This appears to be a more widespread internet issue that is not limited to our platform.<br /><br />Distributed smartlink apps are not currently known to be affected.<br /><br />We will continue to monitor the situation and bring more updates as soon as more information is available.</p>tag:status.kornitx.net,2005:Incident/45762602020-07-09T23:47:36Z2022-02-04T11:40:14ZFirewall Maintanence<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>23:47</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>23:21</var> UTC</small><br><strong>Investigating</strong> - Our hosting partner is currently performing a data centre firewall security update that is taking longer than expected.<br /><br />We are hoping for connectivity to be restored shortly.</p>tag:status.kornitx.net,2005:Incident/44073272020-06-18T16:00:00Z2022-02-04T11:40:14ZSystem Outage<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Resolved</strong> - At approximetly 16:00 UTC our primary database server started to show extremely high CPU load and reached a maximum number of connections allowed at any one time.<br /><br />We immediately stopped all scheduled tasks to try to conserve connections and CPU time for the web tier. This only resulted in a small improvement as the vast majoiry of database traffic from scheduled tasks hits our database replicas rather than the primary however it was a sensible first step.<br /><br />Further investigation revealed the majority of the load was being generated from a particular SQL query within our user authentication mechanism which had started applying significant backpressure on the primary database server due to an unusually large influx of concurrent users trying to login to the system.<br /><br />We subsequently prevented any new user sessions from being initialised and the CPU load quickly reduced back down to acceptable levels.<br /><br />We have now deployed a more efficient version of the SQL query in question.<br /><br />Initial performance testing shows that the new query is several orders of magnitude faster and we do not antipciate any further occurances of this particular problem.</p>