tag:deskprostatus.com,2005:/historyDeskpro Status - Incident History2024-03-28T18:40:37+00:00Deskprotag:deskprostatus.com,2005:Incident/189136622023-10-25T10:12:50+01:002023-10-25T10:12:50+01:00We are investigating connectivity issues affecting some customers<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>10:12</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>16:15</var> BST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>15:56</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:deskprostatus.com,2005:Incident/181150122023-08-11T13:12:54+01:002023-08-11T13:13:15+01:00We are investigating connectivity issues affecting some customers<p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>13:12</var> BST</small><br><strong>Resolved</strong> - We are marking this incident resolved. We are very sorry about the disruption this caused; we continue to work with Cloudflare to prevent this situation from reoccurring.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>13:03</var> BST</small><br><strong>Update</strong> - We are not saying any further impact with full connectivity restored.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>12:47</var> BST</small><br><strong>Monitoring</strong> - Cloudflare have been working to address the issue and we see significantly improved connectivity. We are monitoring.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>12:27</var> BST</small><br><strong>Update</strong> - We are very sorry about the delay in resolving this. Our teams continue to work to restore full service.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>12:14</var> BST</small><br><strong>Identified</strong> - We have identified issues with our CDN Cloudflare. We are working with them to resolve the issue.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>11:59</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:deskprostatus.com,2005:Incident/180315212023-08-02T15:16:03+01:002023-08-02T20:05:25+01:00We are investigating slowness/connectivity issues affecting some customers<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>15:16</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>15:13</var> BST</small><br><strong>Update</strong> - The incident should be resolved. We are very sorry about the disruption caused and are looking into further understanding the initial cause to prevent it from happening again.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>15:08</var> BST</small><br><strong>Update</strong> - We have failed over the affected database cluster to resolve the issue. This will cause connection issues for some customers for a couple of minutes.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>14:49</var> BST</small><br><strong>Update</strong> - We have identified the root cause (increased database load during release migrations) and are mitigating this now.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>14:47</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:deskprostatus.com,2005:Incident/179173832023-07-21T07:20:02+01:002023-07-21T07:20:02+01:00Connectivity Issues<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>07:20</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>01:47</var> BST</small><br><strong>Monitoring</strong> - Service has been restored and we are monitoring connectivity</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>01:32</var> BST</small><br><strong>Investigating</strong> - We are investigating connectivity issues affecting some customers</p>tag:deskprostatus.com,2005:Incident/175132372023-06-08T15:56:34+01:002023-06-08T15:56:34+01:00Connectivity Issues<p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>15:56</var> BST</small><br><strong>Resolved</strong> - We are seeing the affected part of the platform returning to normal and the issue being resolved.</p><p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>15:35</var> BST</small><br><strong>Update</strong> - We are initiating an update to deploy additional resources to a cluster; this may cause 1-2 minutes of downtime</p><p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>15:34</var> BST</small><br><strong>Investigating</strong> - We have been investigating reports of slowness/connectivity issues affecting some customers.</p>tag:deskprostatus.com,2005:Incident/174897712023-06-06T14:40:39+01:002023-06-06T14:40:39+01:00We are investigating connectivity issues with Deskpro<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>14:40</var> BST</small><br><strong>Resolved</strong> - A database failover caused approximately 3 minutes of downtime. We apologies for the disruption and are invesitgating the root cause.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>14:36</var> BST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:deskprostatus.com,2005:Incident/168374082023-04-13T12:52:45+01:002023-04-13T12:54:35+01:00Delays in email processing<p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>12:52</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>17:27</var> BST</small><br><strong>Monitoring</strong> - The backlog has been fully processed and new messages are now processing normally in real time.</p><p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>16:00</var> BST</small><br><strong>Investigating</strong> - We are aware of some customers experiencing delays with email processing. We are investigating.</p>tag:deskprostatus.com,2005:Incident/158359062023-01-13T13:27:46+00:002023-01-13T13:27:46+00:00SSL errors on some custom domains<p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>13:27</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>13:12</var> GMT</small><br><strong>Identified</strong> - We are aware of SSL errors on some cloud accounts using custom domain names.<br /><br />The issue has been identified and we are working on a fix.</p>tag:deskprostatus.com,2005:Incident/144360922022-12-05T21:09:45+00:002022-12-05T21:09:45+00:00AWS internet connectivity issues<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>21:09</var> GMT</small><br><strong>Resolved</strong> - AWS has issued a status update https://health.aws.amazon.com/health/status<br /><br />[RESOLVED] Internet Connectivity<br /><br />[01:06 PM PST] Between 11:34 AM and 12:51 PM PST, customers experienced Internet connectivity issues for some networks to and from the US-EAST-2 Region. Connectivity between instances within the Region, in between Regions, and Direct Connect connectivity were not impacted by this issue. The issue has been resolved and connectivity has been fully restored.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>21:07</var> GMT</small><br><strong>Update</strong> - AWS has issued a status update https://health.aws.amazon.com/health/status<br /><br />[12:59 PM PST] We are beginning to see signs of recovery, and continue to work toward full resolution.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>20:55</var> GMT</small><br><strong>Identified</strong> - AWS has issued a status update https://health.aws.amazon.com/health/status<br /><br />[12:51 PM PST] We can confirm an issue which is impacting Internet connectivity for the US-EAST-2 Region, and are attempting multiple parallel mitigation paths. Connectivity between instances within the US-EAST-2 Region, in-between AWS Regions, and Direct Connect traffic is not impacted by the event. Some customers may be experiencing VPN connectivity due to this issue.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>20:32</var> GMT</small><br><strong>Update</strong> - AWS is aware and investigating https://health.aws.amazon.com/health/status<br /><br />[12:26 PM PST] We are investigating an issue which may be impacting Internet connectivity between some customer networks and the US-EAST-2 Region.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>20:18</var> GMT</small><br><strong>Investigating</strong> - We are investigating reports of connectivity issues in the US region.<br /><br />Update: It appears that Amazon Web Services (AWS) is experiencing issues. We are working to figure out how we can work around these issues.</p>tag:deskprostatus.com,2005:Incident/127666672022-11-10T09:02:09+00:002022-11-10T09:02:09+00:00Agent interface failing to load for some users<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>09:02</var> GMT</small><br><strong>Resolved</strong> - Cloudflare has resolved the incident.<br /><br />If you still experience any issues with assets loading on the login page, please reach out to Deskpro support.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>17:37</var> GMT</small><br><strong>Identified</strong> - We have identified the issue as being a problem with our CDN provider, Cloudflare. They have posted their own status update here:<br /><br />https://www.cloudflarestatus.com/incidents/bvstlwtc0gqj</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>15:43</var> GMT</small><br><strong>Investigating</strong> - We are investigating reports of the agent interface (/app) failing to load for some users due to static assets failing to load from our CDN.</p>tag:deskprostatus.com,2005:Incident/126140232022-10-26T09:01:36+01:002022-10-26T09:03:04+01:00Issues using the Deskpro Interface<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>09:01</var> BST</small><br><strong>Resolved</strong> - Cloudflare is reporting this issue as fully resolved: https://www.cloudflarestatus.com/incidents/kdpqngcbbn25</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>19:33</var> BST</small><br><strong>Monitoring</strong> - Cloudflare has deployed a fix which should resolve all known issues. We will continue to monitor.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>18:29</var> BST</small><br><strong>Identified</strong> - We are aware of some customers having issues logging in or loading various parts of the interface.<br /><br />This is due to a problem with our network provider Cloudflare: https://www.cloudflarestatus.com/incidents/kdpqngcbbn25<br /><br />We are currently monitoring the situation alongside Cloudflare who is already working on a fix.</p>tag:deskprostatus.com,2005:Incident/111231062022-09-13T21:08:23+01:002022-09-13T21:08:23+01:00Investigating Connectivity Issues<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>21:08</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>20:55</var> BST</small><br><strong>Investigating</strong> - We are investigating connectivity issues affecting some Deskpro customers.</p>tag:deskprostatus.com,2005:Incident/110328212022-09-01T16:52:49+01:002022-09-01T16:52:49+01:00Incoming email backlog in the UK1 region<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>16:52</var> BST</small><br><strong>Resolved</strong> - This incident is now resolved and we've seen no problems over the past few hours.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:49</var> BST</small><br><strong>Monitoring</strong> - The backlog is now being processed normally.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>01:30</var> BST</small><br><strong>Investigating</strong> - We are aware of an email backlog affecting some customers in our UK1 region. We are investigating the cause and will update this post when we have more information.</p>tag:deskprostatus.com,2005:Incident/107585972022-08-04T12:58:41+01:002022-08-04T12:58:41+01:00Some emails appearing with blank messages<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>12:58</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>12:17</var> BST</small><br><strong>Monitoring</strong> - Existing messages in tickets should all be showing normally.<br /><br />There is a small number of email messages in the backlog that should be done processing within 10-15 minutes.</p><p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>10:50</var> BST</small><br><strong>Identified</strong> - We're aware of a problem with incoming email messages appearing as blank in the agent interface.<br /><br />We've identified the issue and messages will be properly visible shortly.</p>tag:deskprostatus.com,2005:Incident/107109452022-08-01T11:20:29+01:002022-08-01T11:20:30+01:00Network connectivity issues in AWS<p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>11:20</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>19:42</var> BST</small><br><strong>Monitoring</strong> - We have moved our infrastructure off of the affected AZs and service should now be restored.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>19:01</var> BST</small><br><strong>Update</strong> - We are currently working to move our systems off of the affected availability zone to mitigate the issues.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>18:30</var> BST</small><br><strong>Update</strong> - AWS is working on resolving the issue. More detail can be seen here on AWS's status page: https://health.aws.amazon.com/health/status</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>18:26</var> BST</small><br><strong>Identified</strong> - Our hosting provider, Amazon Web Services (AWS), is experiencing network connectivity issues in our US region. This is causing some web requests to fail to load.<br /><br />We will update this incident once we know more.</p>tag:deskprostatus.com,2005:Incident/103669002022-06-21T11:06:11+01:002022-06-21T11:06:11+01:00Cloudflare Outage<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>11:06</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>08:19</var> BST</small><br><strong>Monitoring</strong> - Cloudflare, a major service provider, experienced a widespread outage this morning resulting in Deskpro failing to load. <br /><br />The issue appears to be resolved and we are monitoring the situation. <br /><br />Read more from Cloudflare here:<br />https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj</p>tag:deskprostatus.com,2005:Incident/103360852022-06-18T10:00:39+01:002022-06-18T10:00:39+01:00Database maintenance in the US region<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>10:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>09:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>11:56</var> BST</small><br><strong>Scheduled</strong> - Deskpro will be conducting database maintenance for sites in the US region. Helpdesks may be unavailable for up 30 seconds during the maintenance window.</p>tag:deskprostatus.com,2005:Incident/96487412022-05-31T16:19:02+01:002022-05-31T16:19:02+01:00Service disruption for some customers<p><small>May <var data-var='date'>31</var>, <var data-var='time'>16:19</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>16:48</var> BST</small><br><strong>Monitoring</strong> - All affected accounts are back online.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>15:04</var> BST</small><br><strong>Identified</strong> - We have identified an issue affecting a small number of users caused by a problem with a specific database cluster. We are moving affected users to a new cluster.</p>tag:deskprostatus.com,2005:Incident/94349932022-03-01T17:24:53+00:002022-03-01T17:24:53+00:00Service disruption for UK Customers<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>17:24</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved. We are very sorry about the disruption caused. We will update with further details shortly.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>14:19</var> GMT</small><br><strong>Identified</strong> - We apologise for the continued service issues. We have identified the issue with failures in AWS Aurora. We are working to resolve this.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>12:57</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating a service disruption affecting some UK customers</p>tag:deskprostatus.com,2005:Incident/90089502022-01-20T09:30:29+00:002022-01-20T09:30:29+00:00US Region Database Maintenance<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>09:30</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>09:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>15:14</var> GMT</small><br><strong>Scheduled</strong> - Deskpro will be conducting scheduled database maintenance on January 20 from 09:00-09:30 UTC for sites in the US region. Helpdesks may be unavailable for up to a minute during the maintenance window.<br /><br />We'll update this status once maintenance has been completed.</p>tag:deskprostatus.com,2005:Incident/90089402022-01-19T18:00:29+00:002022-01-19T18:00:29+00:00EU Region Database Maintenance<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>18:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>17:30</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>15:13</var> GMT</small><br><strong>Scheduled</strong> - Deskpro will be conducting scheduled database maintenance on January 19 from 17:30-18:00 UTC for sites in the EU region. Helpdesks may be unavailable for up to a minute during the maintenance window.<br /><br />We'll update this status once maintenance has been completed.</p>tag:deskprostatus.com,2005:Incident/90089292022-01-18T18:00:29+00:002022-01-18T18:00:29+00:00UK Region Database Maintenance<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>18:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>17:30</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>15:12</var> GMT</small><br><strong>Scheduled</strong> - Deskpro will be conducting scheduled database maintenance on January 18 from 17:30-18:00 UTC for sites in the UK region. Helpdesks may be unavailable for up to a minute during the maintenance window.<br /><br />We'll update this status once maintenance has been completed.</p>tag:deskprostatus.com,2005:Incident/81198242021-11-25T09:02:04+00:002021-11-25T09:02:04+00:00Connectivity Issues<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>09:02</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>11:05</var> BST</small><br><strong>Monitoring</strong> - We have applied a fix, the platform has returned to normal and we continue to monitor.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>11:02</var> BST</small><br><strong>Investigating</strong> - We are investigating connectivity issues affecting some customers</p>tag:deskprostatus.com,2005:Incident/86828092021-11-25T09:01:54+00:002021-11-25T09:01:54+00:00Voice and Incoming Email issues affecting some Cloud Customers<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>09:01</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>20:36</var> GMT</small><br><strong>Monitoring</strong> - The issue with the service provider has been resolved and the Voice service is now restored. Some delays to incoming emails are still to be expected until we finish processing the backlog of messages.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>19:54</var> GMT</small><br><strong>Investigating</strong> - This issue is also causing delays in incoming emails. No data has been lost but some customers may see delays in receiving messages.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>19:39</var> GMT</small><br><strong>Monitoring</strong> - An issue with a third party provider is causing issues for Deskpro Voice. We're monitoring the situation and will post updates when more is known about the problem. For more information, please check https://status.aws.amazon.com/</p>tag:deskprostatus.com,2005:Incident/55776932020-11-16T13:36:00+00:002021-03-12T12:48:01+00:00Connectivity Issues<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>13:36</var> GMT</small><br><strong>Resolved</strong> - This incident has been resolved. We are investigating the delay in connectivity during platform recovery.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>13:27</var> GMT</small><br><strong>Investigating</strong> - We are investigating connectivity issues affecting our EU datacentre</p>