.. _how-to-add-forge-now-pipeline:
Add-forge-now automation
========================
.. admonition:: Intended audience
:class: important
sysadm staff members
When a request status is updated to ``Accepted``, a Gitlab pipeline is triggered to process the request.
* `Add-forge-now Archive `_
* `Add-forge-now-requests Repository `_
* `Add-forge-now-requests CI/CD Analytics `_
.. _add-forge-now-pipeline:
Pipeline
--------
| To overview the pipelines,
see the `pipelines `_ page.
| To see only the running pipelines,
see the `running pipelines `_ page.
| To view the pipeline details, click on the pipeline status.
| To view the job details, click on the stage and choose a job.
.. admonition:: Concurrent jobs
:class: note
| Only ten concurrent pipelines can be processed at the same time.
| It can be changed on ``runner0``:
.. code-block:: console
root@runner0:~# awk '/concurrent/' config.toml
concurrent = 10
|afnr_pipeline|
.. |afnr_pipeline| image:: ../images/add-forge-now/AFNR_pipeline.png
.. _add-forge-now-pre-stage:
Pre stage
---------
| The first job create a Gitlab issue with the forge URL in the title.
| To overview the issues,
see the `issues, `_ page.
| The second job check the SWH components ports,
the forge URL (https) availability and the webapp token validity (recreate it if needed).
| The last job of the ``pre`` stage add a comment on the webapp request with the Gitlab issue URL.
|afnr_pipeline_pre_stage|
.. |afnr_pipeline_pre_stage| image:: ../images/add-forge-now/AFNR_pipeline_pre_stage.png
.. _add-forge-now-staging-stages:
Staging stages
--------------
| The second, third and fourth stages process the request in staging environment.
| If all the jobs are successfully completed, the request status is updated to ``Scheduled`` and
a comment is added to the Gitlab issue.
|afnr_pipeline_staging_stages|
.. |afnr_pipeline_staging_stages| image:: ../images/add-forge-now/AFNR_pipeline_staging_stages.png
.. _add-forge-now-production-stages:
Production stages
-----------------
| The fifth, sixth and seventh stages process the request in production environment.
| If all the jobs are successfully completed, the request status is updated to ``First Origins Loaded``,
a comment is added to the Gitlab issue and this issue is closed.
|pipeline_production_stages|
.. |pipeline_production_stages| image:: ../images/add-forge-now/AFNR_pipeline_production_stages.png
.. _add-forge-now-request-status:
Request status
--------------
* **pre** stage adds a comment on request;
* if the **staging** stages are successfully completed, the request status is updated to ``Scheduled``;
* if the **production** stages are successfully completed, the request status is updated to ``First Origin Loaded``.
|afnr_request_status|
.. |afnr_request_status| image:: ../images/add-forge-now/AFNR_request_status.png
All comments contain the url of the issue where to find the associated pipeline.
|afnr_request_comment|
.. |afnr_request_comment| image:: ../images/add-forge-now/AFNR_request_comment.png
.. _add-forge-now-common-pipeline-errors:
Common errors
-------------
|afnr_errors|
.. |afnr_errors| image:: ../images/add-forge-now/AFNR_errors.png
A non-exhaustive list of common pipeline errors is:
- Stage `pre`, job `02_check_ports_and_token`:
- error: the network ports checks failed;
- cause: one of the SWH components are unreachable or the forge URL (https) is unreachable;
- relaunch the job.
- Stage `*_list_origins`, job `2*_check_listed_origins`:
- error: ``Forge () isn't registered in the scheduler database.``;
- cause: the lister pod creation took more time than the listing delay (600s);
- relaunch the job.
- Stage `*_list_origins`, job `2*_check_listed_origins`:
- error: ``Forge () has 0 listed origin in the scheduler database.``;
- cause: the lister failed to find some origin;
- process request manually (most of the time, it needs the optional `base_git_url` parameter).
See :ref:`How to process add-forge-now requests `
- Stage `*_ingest_origins`, job `4*_check_ingested_origins`:
- error: ``There are too many ingestion failures.``;
- cause: ingestion failures > 70%;
- check ingestion (e.g. lots of empty repositories,...) and/or swh platform (e.g. objstorage unavailable,...);
process request manually (combination of `url` and `base_git_url`).
- Any stage, any job (generally in job `4p_check-ingested-origins`):
- error: ``ERROR: Job failed: execution took longer than 168h0m0s seconds``;
- cause: job timeout (7d);
- relaunch the job.
Gitlab-runner
-------------
| The Gitlab-runner is dedicated to the add-forge-now-requests project.
| The `CI/CD settings `_ page contains:
| - the ``Runners`` section to check the runner's status and update the runner configuration;
|afnr_gitlab-runner|
.. |afnr_gitlab-runner| image:: ../images/add-forge-now/AFNR_gitlab-runner.png
:scale: 50%
| - the ``Variables`` section to check, retrieve or update the variables that need to be protected and masked;
.. admonition:: Webapp Token
:class: note
If the webapp token is expired and not renewed (normally it should), the pipeline will fail in third job of **pre** stage.
You can update the variable ``WEBAPP_TOKEN`` in this section.
.. admonition:: addForgeNowIssue Token
:class: note
This project token is used to create, comment and close the GitLab issues and to commit the request status in the
`requests processing history `_.
The project token can be generated on page `access token `_
and requires an ``api`` scope and a ``maintainer`` role. Once created the value must be stored in the variable ``ADD_FORGE_NOW_ISSUE_TOKEN`` [1]_.
The bot user is automatically created the first time the token is used. Project token has a lifetime of one year.
When the project token has expired, the bot user is deleted [2]_.
| - the ``Pipeline trigger tokens`` section to check or change the pipeline token.
|
| It runs on **runner0** virtual machine with a docker executor.
``docker-compose.yml``
.. code:: yaml
version: "3"
services:
gitlab-runner:
image: gitlab/gitlab-runner:alpine
container_name: gitlab-runner
restart: always
volumes:
- "/run/docker.sock:/var/run/docker.sock"
- "gitlab_gitlab-runner:/etc/gitlab-runner"
- "./config.toml:/etc/gitlab-runner/config.toml"
volumes:
gitlab_gitlab-runner:
Runner configuration file ``config.toml``
.. code:: yaml
concurrent = 10
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "add-forge-now-runner"
url = "https://gitlab.softwareheritage.org"
id = 4
token = "xxx"
token_obtained_at = 2023-10-12T14:12:16Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.docker]
tls_verify = false
image = "ruby:2.7"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache",
"/etc/swh/scheduler-staging.yml:/etc/swh/scheduler-staging.yml",
"/etc/swh/scheduler-production.yml:/etc/swh/scheduler-production.yml"]
shm_size = 0
Check the runner version and status:
.. code:: bash
root@runner0:~# alias dck
alias dck='docker compose'
root@runner0:~# dck exec -ti gitlab-runner gitlab-runner --version
Version: 16.10.0
Git revision: 81ab07f6
Git branch: 16-10-stable
GO version: go1.21.7
Built: 2024-03-21T19:43:25+0000
OS/Arch: linux/amd64
root@runner0:~# dck exec -ti gitlab-runner gitlab-runner verify
Runtime platform arch=amd64 os=linux pid=177 revision=81ab07f6 version=16.10.0
Running in system-mode.
Verifying runner... is valid runner=6aU8maWXg
----
.. [1] Ensure the variable has the ``masked`` attribute.
.. [2] The previous issue comments appeared with ``Ghost User`` author.