Fast cache for node_modules

GitLab is too slow for large repositories including huge number of dependencies (NodeJS projects, usually) and even turning the FF_USE_FASTZIP flag on can't help much. One way to make the build faster is to just remove the cache zipping and upload/download step of them whatsoever.

The trick is to modify the GIT_CLEAN_FLAGS and prevent GitLab to remove node_modules and the --cache-folder used in yarn command (usually .yarn).

variables:
  # ...
  GIT_CLEAN_FLAGS: -ffdx -e **/node_modules -e **/.yarn

Now you have a new problem: your build fails if gitlab puts different jobs in different folders (because the other folder doesn't have the node_modules folder yet). To use a custom static build folder turn on the custom build directory feature for your runner. You runner's config file is most probabely under /etc/gitlab-runner folder, called config.toml or something like that.

[[runners]]
  name = "MyRunner"
  # ...
  [runners.custom_build_dir]
    enabled = true
  
  # ...

And then pick an static build folder for each branch/tag:

variables:
  # ...
  GIT_CLEAN_FLAGS: -ffdx -e **/node_modules -e **/.yarn
  GIT_CLONE_PATH: '$CI_BUILDS_DIR/$CI_PROJECT_PATH_SLUG/$CI_COMMIT_REF_SLUG'

Now you have a third problem to fix. When different jobs are trying to run in the same folder, you might end up facing this error:

fatal: unable to access REPO_URL (path? access rights?)

One solution is to run git clone commands one at a time, so clones of different jobs won't interfere with eachother. The other solution which is fairly easier but slower is to just run those jobs in sequence (using DAG pipelines).

For other cache and artifact folders which are not as huge as node_modules you might want to enable FASTZIP or reduce the compression level:

variables:
  # ...
  GIT_CLEAN_FLAGS: -ffdx -e **/node_modules -e **/.yarn
  GIT_CLONE_PATH: '$CI_BUILDS_DIR/$CI_PROJECT_PATH_SLUG/$CI_COMMIT_REF_SLUG'
  FF_USE_FASTZIP: "true"
  TRANSFER_METER_FREQUENCY: "30s"
  CACHE_COMPRESSION_LEVEL: "fastest"
  ARTIFACT_COMPRESSION_LEVEL: "fastest"