Skip to main content

General

When working with Seqera Platform, you might encounter the following issues.

Common errors​

timeout is not an integer or out of range​

This error occurs on Seqera Platform v24.2 and later when Redis is outdated. Version 24.2 requires Redis 6.2 or later. To resolve, upgrade your Redis instance according to your cloud provider's instructions.

Unknown pipeline repository or missing credentials from public GitHub repositories​

GitHub imposes rate limits on repository pulls, including public repositories: unauthenticated requests are capped at 60 per hour and authenticated requests at 5000 per hour. This error is usually caused by the 60-per-hour cap.

To resolve:

  1. Ensure there's at least one GitHub credential in your workspace's Credentials tab.

  2. Ensure the Access token field of every GitHub credential is populated with a personal access token and not a user password. GitHub personal access tokens (PATs) are typically longer than passwords and include a ghp_ prefix. For example: ghp_IqIMNOZH6zOwIEB4T9A2g4EHMy8Ji42q4HA

  3. Confirm that your PAT provides the elevated threshold and that transactions are charged against it:

    curl -H "Authorization: token ghp_LONG_ALPHANUMERIC_PAT" -H "Accept: application/vnd.github.v3+json" https://api.github.com/rate_limit

No such variable​

This error occurs when you execute a DSL1-based Nextflow workflow with Nextflow 22.03.0-edge or later.

Sleep commands in Nextflow workflows​

The behavior of sleep commands in your Nextflow workflows depends on where they are used:

  • In an errorStrategy block, Nextflow uses the Groovy sleep function, which takes its value in milliseconds.
  • In a process script block, that language's sleep binary or method is used. For example, this bash script uses the bash sleep binary, which takes its value in seconds.

Large number of batch job definitions​

Platform normally looks for an existing job definition that matches your workflow requirement. If nothing matches, it recreates the job definition. Use a bash script to clear job definitions. Tailor it to your needs, for example to deregister only job definitions older than a set number of days:

jobs=$(aws --region eu-west-1 batch describe-job-definitions | jq -r .jobDefinitions[].jobDefinitionArn)

for x in $jobs; do
  echo "Deregister $x";
  sleep 0.01;
  aws --region eu-west-1 batch deregister-job-definition --job-definition $x;
done

Containers​

Use rootless containers in Nextflow pipelines​

Most containers use the root user by default. Some users prefer a non-root user in the container to minimize the risk of privilege escalation. Because Nextflow and its tasks use a shared work directory to manage input and output data, rootless containers can cause file permission errors in some environments:

touch: cannot touch '/fsx/work/ab/27d78d2b9b17ee895b88fcee794226/.command.begin': Permission denied

This should not occur with AWS Batch from Seqera version 22.1.0. In other cases, force all task containers to run as root. Add one of the following to your Nextflow configuration:

// cloud executors
process.containerOptions = "--user 0:0"

// Kubernetes
k8s.securityContext = [
  "runAsUser": 0,
  "runAsGroup": 0
]

Git integration​

Get branches operation not supported by BitbucketServerRepositoryProvider provider​

If you supplied the correct Bitbucket credentials and URL details in your tower.yml and still see this error, upgrade to at least v22.3.0. This version addresses SCM provider authentication issues and likely resolves the retrieval failure.

Optimization​

OutOfMemoryError: Container killed due to memory usage​

Nextflow can underestimate the memory allocation for containerized tasks. As a workaround, add a retry error strategy to the failing process that increases the allocated memory on each retry:

process {
    errorStrategy = 'retry'
    maxRetries    = 3
    memory = { 1.GB * task.attempt }
}

Plugins​

Use the Nextflow SQL DB plugin to query AWS Athena​

From Nextflow 22.05.0-edge, your Nextflow pipelines can query data from AWS Athena. Add these items to your nextflow.config. Secrets are optional:

plugins {
  id 'nf-sqldb@0.4.0'
}

sql {
    db {
        'athena' {
              url = 'jdbc:awsathena://AwsRegion=<region>;S3OutputLocation=s3://<s3-bucket>'
              user = secrets.ATHENA_USER
              password = secrets.ATHENA_PASSWORD
            }
    }
}

Then call the functionality in your workflow:

channel.sql.fromQuery("select * from test", db: "athena", emitColumns:true).view()
note

This example uses the legacy nf-sqldb@0.4.0 syntax. Newer plugin versions use an explicit include { fromQuery } from 'plugin/nf-sqldb' statement instead. See the nf-sqldb documentation.

See the nf-sqldb discussion for more information.

Repositories​

Private Docker registry integration​

Seqera-invoked jobs can pull container images from private Docker registries, such as JFrog Artifactory. The method depends on your computing platform.

For AWS Batch, modify your EC2 launch template using these AWS instructions.

note

This solution requires Docker Engine 17.07 or later to use --password-stdin.

You might need to add commands to your launch template, depending on your security posture:

cp /root/.docker/config.json /home/ec2-user/.docker/config.json && chmod 777 /home/ec2-user/.docker/config.json

For Azure Batch, create a Container registry-type credential in your Seqera workspace and associate it with the Azure Batch compute environment in the same workspace.

For Kubernetes, use an imagePullSecret, per #2827.

Remote resource not found​

This error occurs when the Nextflow head job fails to retrieve the repository credentials from Seqera. If your Nextflow log contains an entry like DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-], check the protocol of your instance's TOWER_SERVER_URL value. It must be set to https rather than http, unless you use TOWER_ENABLE_UNSAFE_MODE to allow HTTP connections to Seqera in a test environment.

Secrets​

Missing AWS execution role arn during launch​

The ECS agent must have access to retrieve secrets from AWS Secrets Manager. Secrets-using pipelines launched in an AWS Batch compute environment encounter this error when an IAM execution role is not provided. See Secrets.

AWS Batch task failures with secrets​

You might encounter errors when executing pipelines that use secrets on AWS Batch:

  • If you use nf-sqldb version 0.4.1 or earlier and have secrets in your nextflow.config, you might see nextflow.secret.MissingSecretException: Unknown config secret errors in your Nextflow log.

    To resolve, explicitly define the xpack-amzn plugin in your configuration:

    plugins {
      id 'xpack-amzn'
      id 'nf-sqldb'
    }

  • If you have two or more processes that use the same container image but only some of them use secrets, your secret-using processes might fail during the initial run and then succeed when resumed. This is caused by a bug in how Nextflow (22.07.1-edge and earlier) registers jobs with AWS Batch.

    To resolve, upgrade Nextflow to version 22.08.0-edge or later. If you can't upgrade, use one of these workarounds:

    • Use a different container image for each process.
    • Define the same set of secrets in each process that uses the same container image.

Tower Agent​

Unexpected Exception in WebSocket … Operation timed out​

Tower Agent reconnection logic was improved in version 0.5.0. Update your Tower Agent before relaunching your pipeline.

Google​

Spot VM preemption causes task interruptions​

Spot VMs reduce cost but increase the likelihood that a task is interrupted before completion. When Google Cloud reclaims a Spot VM, Google Cloud Batch terminates the task with exit code 50001. Add a retry strategy to your Nextflow configuration so interrupted tasks are automatically re-executed. See Spot Instances in the Nextflow documentation. For example:

process {
  errorStrategy = { task.exitStatus == 50001 ? 'retry' : 'finish' }
  maxRetries    = 5
}

Seqera service account permissions for Google Cloud Batch​

Grant the following roles to the custom service account that submits Batch jobs:

  • Batch Agent Reporter (roles/batch.agentReporter)
  • Batch Job Editor (roles/batch.jobsEditor)
  • Logs Writer (roles/logging.logWriter)
  • Logs Viewer (roles/logging.logViewer)
  • Service Account User (roles/iam.serviceAccountUser)
  • Storage Admin (roles/storage.admin), or bucket-level Storage access

For detailed setup instructions, see Service account permissions.

Kubernetes​

Invalid value: "xxx": must be less or equal to memory limit​

This error can occur when you specify a value in the Head Job memory field while creating a Kubernetes-type compute environment.

If you receive an error that includes field: spec.containers[x].resources.requests and message: Invalid value: "xxx": must be less than or equal to memory limit, your Kubernetes cluster might be configured with system resource limits that deny the Nextflow head job's resource request. To isolate the component causing the problem, launch a pod directly on your cluster through your Kubernetes administration solution. For example:

---
apiVersion: v1
kind: Pod
metadata:
  name: debug
  labels:
    app: debug
spec:
  containers:
    - name: debug
      image: busybox
      command: ["sh", "-c", "sleep 10"]
      resources:
        requests:
          memory: "xxxMi" # or "xxxGi"
  restartPolicy: Never

On-premises HPC​

java: command not found​

When submitting jobs to your on-premises HPC (using either SSH or Tower Agent authentication), the following error might appear in your Nextflow logs, even with Java on your PATH environment variable:

java: command not found
Nextflow is trying to use the Java VM defined for the following environment variables:
  JAVA_CMD: java
  NXF_OPTS:

Possible causes:

  1. The queue where the Nextflow head job runs is in a different environment or node than your login node userspace.
  2. If your HPC cluster uses modules, the Java module might not be loaded by default.

To troubleshoot:

  1. Open an interactive session with the head job queue.
  2. Launch the Nextflow job from the interactive session.
  3. If your cluster uses modules, add module load <java-module> in the Advanced options > Pre-run script field when creating your HPC compute environment in Seqera.
  4. If your cluster doesn't use modules, source an environment with Java and Nextflow in the Advanced options > Pre-run script field when creating your HPC compute environment in Seqera.

Pipeline submissions to HPC clusters fail for some users​

Nextflow launcher scripts fail if processed by a non-Bash shell, such as zsh or tcsh. You can identify this problem from these error entries:

  1. Your .nextflow.log contains an error like Invalid workflow status - expected: SUBMITTED; current: FAILED.
  2. Your Seqera Error report tab contains an error like:
Slurm job submission failed
- command: mkdir -p /home//\<username\>//scratch; cd /home//\<username\>//scratch; echo <long-base64-string> | base64 -d > nf-<run-id>.launcher.sh; sbatch ./nf-<run-id>.launcher.sh
- exit   : 1
- message: Submitted batch job <#>

Connect to the head node over SSH and run ps -p $$ to verify your default shell. If you see an entry other than Bash, fix it as follows:

  1. Check which shells are available: cat /etc/shells
  2. Change your shell: chsh -s /usr/bin/bash (the path to the binary might differ, depending on your HPC configuration).
  3. If submissions continue to fail after the shell change, ask your Seqera Platform admin to restart the backend and cron containers, then submit again.
On this Page