[PATCH 0/2] MR538: ci: Prepare the macOS build environment in a dedicated CI job.
It is inefficient to download Wine and build `widl` at each run of the `build-mac` job. And while for `widl` itself that's not a big deal, in the future I'd like to build MoltenVK on the CI, so we don't depend on the version available on the CI runner, which would make each CI run quite heavier. So I came up with an `image` CI job that, similarly to Linux, collect together the required dependencies for running the pipeline. macOS doens't support a real Docker image, so this "image" is just a `.tar.gz` file hosted in the GitLab package registry. There is a little problem with that: in GitLab uploading a new file in the package registry doesn't overwrite a file with the same name already available, even if the latest file is returned anyway. So the old versions basically pile up and have to be manually remove every now and then. Hopefully the environment is only seldom recreated, so the storage usage is not a concern. As usual, pipelines for MRs that change the pipeline image itself are bound to fail. A demo pipeline is https://gitlab.winehq.org/giomasce/vkd3d/-/pipelines/20433. -- https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/538
From: Giovanni Mascellani <gmascellani(a)codeweavers.com> --- gitlab/README | 2 +- gitlab/build.yml | 6 +++--- gitlab/image.yml | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/gitlab/README b/gitlab/README index 310912be2..3b5203233 100644 --- a/gitlab/README +++ b/gitlab/README @@ -5,7 +5,7 @@ vkd3d testing scripts These scripts are used by the GitLab CI feature to automatically run the vkd3d tests on each merge request. -The CI target build-image, in the file image.yml, builds a Docker +The CI target image-linux, in the file image.yml, builds a Docker image based on Debian bookworm with all the packages required for testing, and uploads it to the GitLab container registry. The Docker script is in the file image.docker. diff --git a/gitlab/build.yml b/gitlab/build.yml index 9a19316b5..b87ab9695 100644 --- a/gitlab/build.yml +++ b/gitlab/build.yml @@ -5,7 +5,7 @@ image: $CI_REGISTRY/wine/vkd3d:debian-bookworm interruptible: true needs: - - job: build-image + - job: image-linux optional: true dependencies: [] script: @@ -68,7 +68,7 @@ build-crosstest: image: $CI_REGISTRY/wine/vkd3d:debian-bookworm interruptible: true needs: - - job: build-image + - job: image-linux optional: true dependencies: [] script: @@ -94,7 +94,7 @@ build-crosstest: image: $CI_REGISTRY/wine/vkd3d:debian-bookworm interruptible: true needs: - - job: build-image + - job: image-linux optional: true dependencies: [] script: diff --git a/gitlab/image.yml b/gitlab/image.yml index 45a87770b..cda71df6d 100644 --- a/gitlab/image.yml +++ b/gitlab/image.yml @@ -1,4 +1,4 @@ -build-image: +image-linux: stage: image rules: - if: $CI_PIPELINE_SOURCE == 'push' && $CI_PROJECT_PATH == "wine/vkd3d" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH -- GitLab https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/538
From: Giovanni Mascellani <gmascellani(a)codeweavers.com> --- gitlab/build.yml | 14 +++++--------- gitlab/image.yml | 28 ++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 9 deletions(-) diff --git a/gitlab/build.yml b/gitlab/build.yml index b87ab9695..f440c17fe 100644 --- a/gitlab/build.yml +++ b/gitlab/build.yml @@ -130,7 +130,9 @@ build-mac: exit_codes: - 2 interruptible: true - needs: [] + needs: + - job: image-mac + optional: true dependencies: [] tags: - mac @@ -147,14 +149,8 @@ build-mac: - system_profiler SPSoftwareDataType SPHardwareDataType > artifacts/systeminfo.txt - vulkaninfo > artifacts/vulkaninfo.txt - export PATH="/usr/local/opt/bison/bin:/usr/local/opt/flex/bin:$PATH" - - git clone --depth 1 --branch wine-3.21 https://gitlab.winehq.org/wine/wine.git - - cd wine - - mkdir build - - cd build - - ../configure --enable-win64 --without-x --without-freetype - - make tools/widl/widl - - cd ../.. - - export PATH="$PWD/wine/build/tools/widl:$PATH" + - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/mac/0/image.tar.gz" | tar xzv' + - export PATH="$PWD/image/bin:$PATH" - git rebase $CI_MERGE_REQUEST_DIFF_BASE_SHA --exec ./gitlab/build-mac - if [ -f build_failed ] ; then exit 1 ; fi - if [ -f tests_failed ] ; then exit 2 ; fi diff --git a/gitlab/image.yml b/gitlab/image.yml index cda71df6d..d82d5996e 100644 --- a/gitlab/image.yml +++ b/gitlab/image.yml @@ -18,3 +18,31 @@ image-linux: - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf "%s:%s" "$CI_REGISTRY_USER" "$CI_REGISTRY_PASSWORD" | base64 | tr -d '\n')\"},\"$CI_DEPENDENCY_PROXY_SERVER\":{\"auth\":\"$(printf "%s:%s" "$CI_DEPENDENCY_PROXY_USER" "$CI_DEPENDENCY_PROXY_PASSWORD" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json - sed -i "1iFROM $IMAGE_SOURCE" "$DOCKER_FILE" - /kaniko/executor --context "$CI_PROJECT_DIR" --dockerfile "$DOCKER_FILE" --destination "$IMAGE_LOCAL" + +image-mac: + stage: image + rules: + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_PROJECT_PATH == "wine/vkd3d" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + changes: + - .gitlab-ci.yml + - gitlab/image.yml + interruptible: true + tags: + - mac + script: + - mkdir image + - cd image + - mkdir bin + - export PATH="/usr/local/opt/bison/bin:/usr/local/opt/flex/bin:$PATH" + - git clone --depth 1 --branch wine-3.21 https://gitlab.winehq.org/wine/wine.git + - cd wine + - mkdir build + - cd build + - ../configure --enable-win64 --without-x --without-freetype + - make tools/widl/widl + - cd ../.. + - cp wine/build/tools/widl/widl bin + - rm -fr wine + - cd .. + - tar czfv image.tar.gz image + - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file image.tar.gz "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/mac/0/image.tar.gz"' -- GitLab https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/538
As usual, pipelines for MRs that change the pipeline image itself are bound to fail. A demo pipeline is https://gitlab.winehq.org/giomasce/vkd3d/-/pipelines/20433.
Ah, in this case it actually passes, because the package registry is bound to the pipeline namespace rather than being hardcoded to `wine/vkd3d`, so it picks up the package I created for testing. -- https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/538#note_56690
This merge request was approved by Giovanni Mascellani. -- https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/538
This merge request was approved by Henri Verbeet. -- https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/538
It doesn't seem right to abuse the package registry for this. I expect that it should be possible to fetch the artifacts directly from the last image-mac job. -- https://gitlab.winehq.org/wine/vkd3d/-/merge_requests/538#note_56935
participants (4)
-
Alexandre Julliard (＠julliard) -
Giovanni Mascellani
-
Giovanni Mascellani (＠giomasce)
-
Henri Verbeet (＠hverbeet)