Live Status\Incident History

December 14, 2023

Intermittent unavailability of control panel

Degraded Performance

Opened: Dec 14, 2023, 12:26 PM UTC

Duration: 41m

Affected services:Control Panel


  • Investigating
    Dec 14, 2023, 12:26 PM UTC

    Login issue reported, investigation started

  • Identified
    Dec 14, 2023, 12:45 PM UTC

    Cause identified, fix being applied

  • Resolved
    Dec 14, 2023, 1:07 PM UTC

    All services back to normal

December 12, 2023

Checkout offline

Outage

Opened: Dec 12, 2023, 4:24 PM UTC

Duration: 4h 11m

Affected services:Ticket shop


  • Investigating
    Dec 12, 2023, 4:24 PM UTC

    All check-outs appear to be fail during payment.

  • Identified
    Dec 12, 2023, 5:30 PM UTC

    Issue identified, applying fix

  • Resolved
    Dec 12, 2023, 8:35 PM UTC

    Fix successful, all systems online again.

November 30, 2023

Ideal payment not working

Degraded Performance

Opened: Nov 30, 2023, 9:38 AM UTC

Duration: 15m

Affected services:Ticket shop


  • Investigating
    Nov 30, 2023, 9:38 AM UTC

    High error rate in processing IDEAL payments.

  • Identified
    Nov 30, 2023, 9:43 AM UTC

    Issue identified: Rabobank is facing issues processing IDEAL payments.

  • Resolved
    Nov 30, 2023, 9:53 AM UTC

    IDEAL payments are fully functional again.

September 20, 2023

Check-out offline

Outage

Opened: Sep 20, 2023, 1:36 PM UTC

Duration: 5m

Affected services:Ticket shop


  • Identified
    Sep 20, 2023, 1:36 PM UTC

    All check-outs appear offline.

  • Resolved
    Sep 20, 2023, 1:41 PM UTC

    All check-outs back online.

September 4, 2023

Sales reporting not up to date

Missing Data

Opened: Sep 4, 2023, 7:15 AM UTC

Duration: 30h 32m


  • Investigating
    Sep 4, 2023, 7:15 AM UTC

    The sales report function in control panel is not fetching any new data. All other functions are operational.

  • Identified
    Sep 4, 2023, 11:15 AM UTC

    Root cause of the problem has been identified: slow processing data warehouse.

  • Monitoring
    Sep 5, 2023, 6:12 AM UTC

    Hot fix is currently in development. We estimate the sales report coming back online by the end of the day.

  • Resolved
    Sep 5, 2023, 1:47 PM UTC

    Issue has been resolved. All reporting functions are back operational.

July 13, 2023

Servers under heavy load

Degraded Performance

Opened: Jul 13, 2023, 4:33 AM UTC

Duration: 32m

Affected services:Ticket shop


  • Investigating
    Jul 13, 2023, 4:33 AM UTC

    Some user activities, such as viewing products, adding items to the cart, or making payments are experiencing errors.

  • Resolved
    Jul 13, 2023, 5:05 AM UTC

    To resolve the issue, our technical team intervened by manually disabling a service that processes background tasks, fixing the task queue and re-enabling the task service. This action helped stabilize the system and restore the smooth operation of services. We are committed to preventing similar occurrences in the future.

June 2, 2023

Caching service overload

Outage

Opened: Jun 2, 2023, 7:12 AM UTC

Duration: 0s

Affected services:Ticket shop


  • Identified
    May 31, 2023, 9:32 AM UTC

    We're manually adjusting the scale of the caching service. Systems should be back on track soon.

  • Resolved
    May 31, 2023, 9:50 AM UTC

    Systems are now stabilized and operation of services are smooth again. Incident description: During a service disruption, some users might have encountered issues during the checkout process. A significant portion of user activities, such as viewing products, adding items to the cart, or making payments, experienced errors. It is estimated that this affected approximately 8.3% of all requests made within the time period of the disruption. Why did it happen: The service disruption was primarily caused by issues with how our caching service was configured. Certain services were unable to write to the cache as intended, necessitating a direct load of data from the database for each user request. This unexpected burden led to high load levels on our database and subsequent slowdown of services, ultimately causing portion of them to fail.