Dashboards

Public

• https://grafana.f3.eng.filoz.org/public/mainnet
• https://grafana.f3.eng.filoz.org/public/calibnet

Internal versions

• mainnet: https://grafana.f3.eng.filoz.org/d/edsu1k5s7gtf/f3-public-mainnet?orgId=1&refresh=1m
• calibnet: https://grafana.f3.eng.filoz.org/d/edsu1k5s7ggfkf/f3-public-calibnet?orgId=1&refresh=1m
• Starboard F3 dashboard: https://filecoin-dashboard.starboard.ventures/d/edsu1k5s7gtfkb/f3

🔥 Disaster Handling Runbooks

Start an incident

• [ ] In Slack go to the #fil-alerts-confidential channel:

  • [ ] Type /report and click the Report Incident workflow:

    

  • [ ] Set Incident Name to the issue encountered

  • [ ] Set Severity to either P0 or P1 if you need to page in all F3 members.

    • [ ] Set Severity to P2 or P3 if you only need to page in the F3 members that are awake/on-call.

  • [ ] Select F3 in category box - this will notify the F3 team first.

  • [ ] In the Description write a short description about the issue we are investigating.

  After putting in all of the above information, click submit which will trigger an OpsGenie alert to all the F3-members, auto-create a Google Incident Doc in the #fil-alerts

Execute a rapid network upgrade

TODO: fill this in as part of https://github.com/orgs/filecoin-project/projects/114/views/2?pane=issue&itemId=103600326

Inform the community about recommended action of disabling F3

In the event that the best course of action for the network is to disable F3:

1. Identify a canonical location for instructions on how to disable F3 and provide answer to why we’re recommending this action.

   1. Requirements:
      1. Publicly accessible from a browser without any special login credentials.
      2. Disablement instructions won’t get covered over by followup questions
   2. These requirements effectively mean we shouldn’t be relying on slack to post this info. We’ll post to somewhere else and can link to it from Slack.
   3. Since asking the whole community to disable F3 would be a significant wide reaching ask, it’s appropriate to post this in filecoin-project/community.

2. Create issue with instructions and justification.

   Potential text:

   Title: 🚨 Disabling F3 ASAP on YYYY-MM-DD

   Description

   <aside>

   Because of X, we advise the community to disable F3 ASAP. The steps for doing this depend on your implementation.

   Lotus:

   // If before F3 delegated authority contract activation use these steps and remove the other steps. Also delete this line.

   • Set LOTUS_DISABLE_F3_ACTIVATION="contract:[0xA19080A1Bcb82Bb61bcb9691EC94653Eb5315716](<https://filecoin.blockscout.com/address/0xA19080A1Bcb82Bb61bcb9691EC94653Eb5315716?tab=contract>)" (TODO before posting: confirm this value is right - it should match F3ParamsAddress in the release branch.)

   • Restart your node.

   • Confirm you see “F3 activation disabled by environment configuration for contract” in your logs.

     // If after F3 delegated authority contract activation and instead bootstrap was set by a static manifest bootstrap epoch then use these steps and remove the other steps. Also delete this line.

   • Set LOTUS_DISABLE_F3_ACTIVATION="epoch:#########" (TODO before posting: Fill this in based on the F3BootstrapEpoch in the release branch.)

   • Restart your node.

   • Confirm you see “F3 activation disabled by environment configuration for bootstrap epoch” in your logs.

   Forest:

   • Set environment variable FOREST_F3_SIDECAR_FFI_ENABLED=0 to disable F3.
     • There are other possible knobs, outlined in the variables reference, but only the FOREST_F3_SIDECAR_FFI_ENABLED is required for disabling F3.

   Venus:

   • Set environment variable VENUS_DISABLE_F3=1 to disable F3.

   Discussion about when/how F3 should be reenabled will be posted here, or at least linked from here when they are known.

   </aside>

3. Get the message out

   Post a link to the created issue in these places:

   • #fil-network-announcements
   • #fil-implementers
   • #fil-help
   • #fil-fast-finality