Dashboards

Public dashboards from a FilOz-F3-implementer node:

• https://grafana.f3.eng.filoz.org/public/mainnet
• https://grafana.f3.eng.filoz.org/public/calibnet

Internal versions

• mainnet: https://grafana.f3.eng.filoz.org/d/edsu1k5s7gtf/f3-public-mainnet?orgId=1&refresh=1m
• calibnet: https://grafana.f3.eng.filoz.org/d/edsu1k5s7ggfkf/f3-public-calibnet?orgId=1&refresh=1m
• Starboard F3 dashboard: https://filecoin-dashboard.starboard.ventures/d/edsu1k5s7gtfkb/f3

Metrics Monitoring also provides info on how node operators can expose/view their own dashboard concerning F3 performance.

Supporting Infrastructure

FilOz runs the following infrastructure to support operations:

• Grafana dashboard: https://grafana.f3.eng.filoz.org
• f3-observer instance for collecting messages which is useful for analytics and f3-aider instance (discussed below).
  • Monitoring of this process is described in https://github.com/filecoin-project/go-f3/issues/1041
• f3-aider to support the network as a result of https://github.com/filecoin-project/lotus/issues/13094 and https://github.com/filecoin-project/lotus/issues/13208.

Configuration of this infra lives in the private repo https://github.com/filecoin-project/filoz-infra.

🔥 Disaster Handling Runbooks

Start an incident

• [ ] In Slack go to the #fil-alerts-confidential channel:

  • [ ] Type /report and click the Report Incident workflow:

    

  • [ ] Set Incident Name to the issue encountered

  • [ ] Set Severity to either P0 or P1 if you need to page in all F3 members.

    • [ ] Set Severity to P2 or P3 if you only need to page in the F3 members that are awake/on-call.

  • [ ] Select F3 in category box - this will notify the F3 team first.

  • [ ] In the Description write a short description about the issue we are investigating.

  After putting in all of the above information, click submit which will trigger an OpsGenie alert to all the F3-members, auto-create a Google Incident Doc in the #fil-alerts

Execute a rapid network upgrade