This post is about attempt to upgrade the GitLab server installation from version 8.x to 10.x. I explained the issues and how I solved them.
Tldr? Move to the solution.
If you have GitLab server set up, then visit https://URL_to_your_gitlab/help (your own GitLab server installation). It’ll show some info about your GitLab server like it’s version etc. The server I got my hands on hasn’t been upgraded for a while now. Hence, many packages were out of date. So now while running the “apt upgrade” command, it started to upgrade the packages as usual. It downloaded gitlab-ce latest version 10.x.x as expected. Current version was 8.x.x. So when the package-manager reached to the point of setting up gitlab-ce, it started with the configuration. Everything was going well. Then it got to upgrading Postgresql database server. It also needed an upgrade. So usually it migrates the data/tables as per the new structure introduced in newer version. But while configuring it gave errors.
Checking PostgreSQL executables: OK Shutting down all GitLab services except those needed for migrations ok: down: gitlab-workhorse: 4120s, normally up ok: down: logrotate: 4120s, normally up ok: down: nginx: 4119s, normally up ok: down: postgresql: 0s, normally up, want up ok: down: redis: 0s, normally up ok: down: sidekiq: 4119s, normally up ok: down: unicorn: 4119s, normally up timeout: down: postgresql: 0s, normally up, want up ok: run: redis: (pid 48594) 0s down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38594s down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38595s down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38596s down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38597s down: postgresql: 1s, normally up, want up; run: log: (pid 1075) 38598s . . .
and at the end:
Errors were encountered while processing: gitlab-ce
Or even after manually starting postgresql it didn’t show up:
$ sudo gitlab-ctl start postgresql timeout: down: postgresql: 0s, normally up, want up
$ sudo gitlab-ctl status run: gitlab-workhorse: (pid 61389) 1255s; run: log: (pid 1073) 50380s run: logrotate: (pid 61358) 1267s; run: log: (pid 1069) 50380s run: nginx: (pid 61045) 1481s; run: log: (pid 1076) 50380s down: postgresql: 0s, normally up, want up; run: log: (pid 64776) 187s run: redis: (pid 60439) 1730s; run: log: (pid 1071) 50380s run: sidekiq: (pid 65362) 2s; run: log: (pid 1070) 50380s run: unicorn: (pid 65335) 10s; run: log: (pid 1068) 50380s\
Running sudo gitlab-ctl reconfigure showed this error:
$ sudo gitlab-ctl reconfigure PG::ConnectionBad: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.xxxx"?
So it was definitely something wrong with the Postgresql shipped with the gitlab.
I tried to create backup with the following command, but it failed.
$ sudo gitlab-rake gitlab:backup:create rake aborted! storage "default" is missing a gitaly_address
While checking the version info, it showed:
sudo apt-get install nodejs
After installing nodejs, sudo gitlab-rake gitlab:env:info showed some info, but not about the gitlab yet.
$ sudo gitlab-rake gitlab:env:info System information System: Ubuntu 14.04 Current User: git Using RVM: no Ruby Version: 2.3.5p376 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.4 Go Version: unknown rake aborted! PG::ConnectionBad: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.XXXX"?
One may check about the package version using:
sudo apt-cache policy gitlab-ce | grep Installed Installed: 10.3.3-ce.0
Now the main issue left was that of postgresql. So searched a lot about it too.
In gitlab postgres logs at /var/log/gitlab/postgresql/current, I found entries like this:
LOG: unrecognized configuration parameter "unix_socket_directory" in file "/var/opt/gitlab/postgresql/data/postgresql.conf"
I thought of restoring the backup of the GitLab. GitLab backups are stored at /var/opt/gitlab/backups
I tried the following command for restore. Here 1113118501 is the timestamp of the backup file present at /var/opt/gitlab/backups e.g. 1113118501_gitlab_backup.tar
$ sudo gitlab-rake gitlab:backup:restore BACKUP=1113118501 Unpacking backup ... done GitLab version mismatch: Your current GitLab version (10.3.3) differs from the GitLab version in the backup! Please switch to the following version and try again: version: 8.6.5 Hint: git checkout v8.6.5
Two versions of PG were included in GitLab through the 9.x releases, but we removed the older one in 10.0 since it’s been deprecated for a while.
So for installing 9.x, I went to this link and opened 9.5.9 for my distro:
It mentions the steps to install this deb.
The two commands are:
$ curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash Detected operating system as Ubuntu/trusty. Checking for curl... Detected curl... Running apt-get update... done. Installing apt-transport-https... done. Installing /etc/apt/sources.list.d/gitlab_gitlab-ce.list...done. Importing packagecloud gpg key... done. Running apt-get update... done. The repository is setup! You can now install packages.
$ sudo apt-get install gitlab-ce=9.5.9-ce.0 dpkg: warning: downgrading gitlab-ce from 10.3.3-ce.0 to 9.5.9-ce.0 (Reading database ... 110272 files and directories currently installed.) Preparing to unpack .../gitlab-ce_9.5.9-ce.0_amd64.deb ... gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!) Dumping database ... Dumping PostgreSQL database gitlabhq_production ... pg_dump: [archiver (db)] connection to database "gitlabhq_production" failed: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"? [FAILED] Backup failed gitlab preinstall: gitlab preinstall: Backup failed! If you want to skip this backup, run the following command and gitlab preinstall: try again: gitlab preinstall: gitlab preinstall: sudo touch /etc/gitlab/skip-auto-migrations gitlab preinstall: dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_9.5.9-ce.0_amd64.deb (--unpack): subprocess new pre-installation script returned error exit status 1 Errors were encountered while processing: /var/cache/apt/archives/gitlab-ce_9.5.9-ce.0_amd64.deb E: Sub-process /usr/bin/dpkg returned an error code (1)<span data-mce-type="bookmark" id="mce_SELREST_start" data-mce-style="overflow:hidden;line-height:0" style="overflow:hidden;line-height:0" ></span>
$ sudo touch /etc/gitlab/skip-auto-migrations
Checking PostgreSQL executables: OK
Found /etc/gitlab/skip-auto-migrations, exiting…
This time it skipped the database migrations.
Then I executed:
$ sudo gitlab-ctl reconfigure Running handlers complete Chef Client finished, xxx/xxx resources updated in 01 minutes 00 seconds gitlab Reconfigured!
Restarted gitlab and this time everything was fine.
$ sudo gitlab-ctl restart ok: run: gitaly: (pid 107106) 1s ok: run: gitlab-monitor: (pid 107122) 0s ok: run: gitlab-workhorse: (pid 107125) 1s ok: run: logrotate: (pid 107132) 0s ok: run: nginx: (pid 107138) 1s ok: run: node-exporter: (pid 107147) 0s ok: run: postgres-exporter: (pid 107187) 0s ok: run: postgresql: (pid 107195) 0s ok: run: prometheus: (pid 107203) 1s ok: run: redis: (pid 107211) 0s ok: run: redis-exporter: (pid 107215) 0s ok: run: sidekiq: (pid 107220) 1s ok: run: unicorn: (pid 107227) 0s
It includes some new modules too, like gitaly. It was missing earlier that’s why it was giving error related to gitaly earlier (mentioned above). But now everything was fine and GitLab was up and running with a new look and feel.
Now time to upgrade to 10.x.
I thought of upgrading postgresql first.
$ sudo gitlab-ctl pg-upgrade
and it completed with the upgrade. Then I executed:
$ sudo apt upgrade Unpacking gitlab-ce (10.3.4-ce.0) over (9.5.9-ce.0) ... Upgrade complete! If your GitLab server is misbehaving try running sudo gitlab-ctl restart
I restarted as said above some services didn’t start. So I reconfigured again.
$ sudo gitlab-ctl reconfigure
and did some database migrations again. And restarted again and all worked perfectly.
Thanks for reading.