Downtime postmortem 24/04

  • Hello Ahaus,

    We'd like to report an incident that occurred today (Wednesday, April 24th) that, as many of you noticed, left several servers across the board not accessible for hours. The incident started at approximately 6:00 CEST when a configuration error replicated across several servers of Tentlan. The issue was detected early, but unfortunately tracking down the source to ensure we had a solid understanding of the problem and a reliable, durable fix, ended up taking many hours of work, during which many servers unfortunately remained inaccessible. The following servers were affected:

    • Spanish Lakamha (es18)
    • Spanish Ixtlan (es19)
    • German Iximche (de12)
    • German Balamku (de13)
    • German Ixlu (de14)
    • German Tulum (dem2)
    • English Ixlu (en14)
    • English Witzna (en15)
    • French Witzna (fr15)
    • Tenochtitlan (int1)
    • Kantetul (int10)
    • Edzna (int2)
    • Sayil (int4)
    • Miktlan (int8)
    • Becan (intm10)
    • Kinichna (intm11)
    • Tohopku (intm13)
    • Uxul (steam2)
    • Muyil (steam3)

    All these servers are now back online. We have since identified the root cause and applied measures to all servers to prevent this from ever happening again. To compensate for the downtime, we have enabled a global attack protection for anybody playing in the aforementioned servers that will extend to 24 hours from roughly the time of this post.

    We are deeply sorry for the trouble this may have caused, and we hope that you understand that we've worked around the clock to solve this issue despite the several hour long lack of access. We trust that this issue has been fully addressed and will not reoccur in the future.

    Thank you for your understanding and best regards,

    Your Tentlan Team