Live Status\Incident History
December 14, 2023
Intermittent unavailability of control panel
Degraded Performance
Opened: Dec 14, 2023, 12:26 PM UTC
Duration: 41m
Affected services:Control Panel
- InvestigatingDec 14, 2023, 12:26 PM UTC
Login issue reported, investigation started
- IdentifiedDec 14, 2023, 12:45 PM UTC
Cause identified, fix being applied
- ResolvedDec 14, 2023, 1:07 PM UTC
All services back to normal
December 12, 2023
Checkout offline
Outage
Opened: Dec 12, 2023, 4:24 PM UTC
Duration: 4h 11m
Affected services:Ticket shop
- InvestigatingDec 12, 2023, 4:24 PM UTC
All check-outs appear to be fail during payment.
- IdentifiedDec 12, 2023, 5:30 PM UTC
Issue identified, applying fix
- ResolvedDec 12, 2023, 8:35 PM UTC
Fix successful, all systems online again.
November 30, 2023
Ideal payment not working
Degraded Performance
Opened: Nov 30, 2023, 9:38 AM UTC
Duration: 15m
Affected services:Ticket shop
- InvestigatingNov 30, 2023, 9:38 AM UTC
High error rate in processing IDEAL payments.
- IdentifiedNov 30, 2023, 9:43 AM UTC
Issue identified: Rabobank is facing issues processing IDEAL payments.
- ResolvedNov 30, 2023, 9:53 AM UTC
IDEAL payments are fully functional again.
September 20, 2023
Check-out offline
Outage
Opened: Sep 20, 2023, 1:36 PM UTC
Duration: 5m
Affected services:Ticket shop
- IdentifiedSep 20, 2023, 1:36 PM UTC
All check-outs appear offline.
- ResolvedSep 20, 2023, 1:41 PM UTC
All check-outs back online.
September 4, 2023
Sales reporting not up to date
Missing Data
Opened: Sep 4, 2023, 7:15 AM UTC
Duration: 30h 32m
- InvestigatingSep 4, 2023, 7:15 AM UTC
The sales report function in control panel is not fetching any new data. All other functions are operational.
- IdentifiedSep 4, 2023, 11:15 AM UTC
Root cause of the problem has been identified: slow processing data warehouse.
- MonitoringSep 5, 2023, 6:12 AM UTC
Hot fix is currently in development. We estimate the sales report coming back online by the end of the day.
- ResolvedSep 5, 2023, 1:47 PM UTC
Issue has been resolved. All reporting functions are back operational.
July 13, 2023
Servers under heavy load
Degraded Performance
Opened: Jul 13, 2023, 4:33 AM UTC
Duration: 32m
Affected services:Ticket shop
- InvestigatingJul 13, 2023, 4:33 AM UTC
Some user activities, such as viewing products, adding items to the cart, or making payments are experiencing errors.
- ResolvedJul 13, 2023, 5:05 AM UTC
To resolve the issue, our technical team intervened by manually disabling a service that processes background tasks, fixing the task queue and re-enabling the task service. This action helped stabilize the system and restore the smooth operation of services. We are committed to preventing similar occurrences in the future.
June 2, 2023
Caching service overload
Outage
Opened: Jun 2, 2023, 7:12 AM UTC
Duration: 0s
Affected services:Ticket shop
- IdentifiedMay 31, 2023, 9:32 AM UTC
We're manually adjusting the scale of the caching service. Systems should be back on track soon.
- ResolvedMay 31, 2023, 9:50 AM UTC
Systems are now stabilized and operation of services are smooth again. Incident description: During a service disruption, some users might have encountered issues during the checkout process. A significant portion of user activities, such as viewing products, adding items to the cart, or making payments, experienced errors. It is estimated that this affected approximately 8.3% of all requests made within the time period of the disruption. Why did it happen: The service disruption was primarily caused by issues with how our caching service was configured. Certain services were unable to write to the cache as intended, necessitating a direct load of data from the database for each user request. This unexpected burden led to high load levels on our database and subsequent slowdown of services, ultimately causing portion of them to fail.