GitLab Outage
There was a small out of our GitLab instance from 15:08 to 15:40. Read on for post mortem.
News from Jul 29, 2021
Yesterday, we migrated our GitLab instance from version 13.12 on a server running Debian buster to 14.1 on a new server running Debian bullseye. Major migrations like this one take usually take a bit of time because of everything that has bindings against the C-library needs to be recompiled.
GitLab itself consists of multiple components that are mostly written in Ruby and Go, but the Ruby parts use a lot of Gems that use C bindings. While rebuilding everything yesterday we did not fully rebuild Gitaly - the component handling the repositories -, since it's partly written in Go. Some parts, though, are written in Ruby, using some Gems with C-bindings, which we missed, since errors only triggered when showing the project overview page of a project with a license file, which we did not test in our test environment prior to upgrading.
We have now rebuilt Gitaly and everything should be working again. User data was not affected.