Chainlink Labs

Incident Responder Blockchain Network and Systems Administration

Job Description

Posted on: 
2023-04-18 09:22

We’re seeking an Incident Responder (Blockchain Network & Systems Administrator) with knowledge of Blockchains, Smart Contracts, Web2 concepts, networks/network protocols, automation methods, and contingency operations, to perform in-depth troubleshooting of all issues that arise.

In this role you will be ensuring that Chainlink Labs and their products/services remain operational at all times, through both proactive and reactive response to Incidents as well as through leading the Postmortem Process and completion of corrective and preventative actions identified within it.

The primary function in this role is to triage and bring all Incidents to resolution, whether independently or by acting as an Incident Commander, when alerted by our monitoring or other sources. When not actively triaging incidents, you will be working towards the improvement of the Incident Response Process by contributing to the below:

  • Identifying needed policy and procedure changes.
  • Developing contingency plans.
  • Improving the gathering and presentation of Incident related data.
  • Working with other teams to eliminate tech debt which might result in Incidents.
  • Creating automations for common Incident Response tasks.

In addition to the above, as an Incident Responder for the Incident Response Team you will:

  • Respond to all alerts routed to the Incident Response Team, generated from within our monitoring stack, within 1 minute.
  • Evangelize and enact best practices, to guide high-quality Incident Response Process utilization within Chainlink Labs.
  • Create and execute contingency planning exercises, to ensure continued operational readiness of the Incident Response Team, and those we support.
  • Identify and make useful metrics based off of Incident occurrence.


  • Able to work within a 5 days per week shift rotation, within the hours of 2300-0700UTC.
  • Excellent verbal and written English communication skills.
  • Ability to function as a leader (Incident Commander), to both SMEs and leadership, during Major Incidents.
  • Ability to identify risks/issues and develop recommendations for solutions.
  • Familiarity with Git and Infrastructure as code.
  • 3+ years of relevant experience. You may have worked in Network & Systems Administration, Incident Response, Infrastructure or Platform support, Technical Support or other functions.
  • Ability to work in a fast paced environment with dynamic priority evolution.
  • Flexibility to join teamwide meetings, which may be outside of your defined schedule.

Desired Qualifications

  • Ability to program in Python or Go.
  • Experience with distributed systems and container orchestration. You have maintained or even built Kubernetes clusters before and feel comfortable deploying complete new services on them
  • Experience with AWS, Terraform/Terragrunt, Kubernetes, ArgoCD, Prometheus and Grafana, and GitHub Actions.
  • Experience running any infrastructure in the blockchain/web3 space
  • Technical proficiency with Layer 1 and Layer 2 Blockchains.
Apply now

More job openings

By clicking “Accept”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.