Make plan for FilOz-owned/affiliated infra for 2026Q1

[BigLep]

Done Criteria

We have a document (.md file, Notion document, github issue) that outlines all the infra we currently "have", where it is "homed" (e.g., AWS, Hetzner), and where we'd ideally like it. We should also add "rows" for other infrastructure that we see upcoming with the FOC GA launch.

Why Important

As we move to running more infrastructure as part of FOC (e.g., dealbot, FOC explorers), we need to have good operational posture for running those services. Given our smaller team size and not being "super infra versed", it will put less burden on the team to have a more unified strategy than each service being a bespoke solution, on a different provider, with different access patterns.

User/Customer

FilOz team members that will ultimate be on the hook for maintaining the infra.

Notes

1. This issue is in https://github.com/FilOzone/tpm-utils because the repo is public and FilOz tpms will likely be involved... https://github.com/filecoin-project/filoz-infra is private for example.
2. Our plan here should optimize for saving developer time more than monthly costs. We don't want to burn cash unnecessarily, but we aren't a team with a lot of infra experience. We know infra is important for products that real people use (which is a goal of FOC), but because we aren't running infrastructure day-in day-out for our full work days, this isn't a strongly developed muscle. As a result, it's important that we have a solution that makes it easy for:
   1. FilOz to jump in and out of - complexity with lots of layers of abstraction would probably be bad because we'll have to reload all that context in our heads each time we engage.
   2. (related) For AI tooling to give guidance / debug support in - this means using common well established patterns/services so AI models will know how to navigate it, spot issues, etc.
   3. (in future maybe) An "infra-strong" developer or team to come in and support - For example, if we hired someone to help bolster our "devops" muscle, they should be able to look at what we have and not feel like everything needs to be thrown out.
3. We should be pragmatic. We shouldn't disqualify centralized cloud providers (e.g., AWS, GCP) because they're "centralized". If they will help us get the job done more efficiently so we can spend more time building the decentralized FOC we're shooting for, then that's totally fine. (It's fine though to disqualify them for other reasons like lacking features we need or being overly complicated.)

[Kubuxu]

AWS (or another hyperscaler) is an obvious initial choice for a system like this. Still, I’d like us to evaluate the reasons for considering AWS before committing significant resources to learning the intricacies of such a complex ecosystem.

The primary component we’re looking for is a container runtime capable of supporting our workload. This includes running apps like Dealbot, Lotus, custom observation infrastructure, and developer boxes. The spectrum from lightweight apps to Lotus makes this difficult, as Lotus’s resource usage is very high and has previously discouraged running it in EKS or similar AWS setups.

We should definitely level up on the DevOps side, which almost certainly means standardizing on Kubernetes with GitOps. This might seem like a leap, but it’s hard to see a practical alternative. Kubernetes gives us a standardized way to structure, manage, and run containers and their dependencies, ranging from Lotus itself to our applications.

That leads to the question: where should it run? Personally, I’d prefer to avoid a hyperscaler because of the high complexity and potential lock-in. At the same time, we should not manage Kubernetes ourselves; the overhead would be too high for our scale.

Fortunately, we aren’t unique in this need. Managed Kubernetes offerings exist. One provider I’ve encountered repeatedly during my research is Syself. In my opinion, it (or a similar solution) strikes the right balance, offering the standardization and operational simplicity we need without the added complexity of a hyperscaler ecosystem. It also allows us to run bare-metal workloads, such as Lotus, through Kubernetes.

I know the container runtime is only part of the picture. Infrastructure around it, such as metrics, monitoring, and access control, matters too. Fortunately, these are largely standardized within the Kubernetes ecosystem. Using a hyperscaler for this layer, though, often means depending on their specific versions of these tools, which can make it difficult for other teams to use our stack in the future.

[SgtPooki]

I think we might want something even more hands-off than even syself for most things. A managed service like render.com or railway seem very appealing for all non-lotus use-cases.

If we are optimizing for dev time, the more hands off we can get, the better.. and I think migrating everything to kubernetes is ideal in the long term, but maybe too time-consuming for the short term?

I think we should use a managed grafana + prometheus service: grafana cloud, and then from there, VPS/kubernetes-cluster/etc.. should really be anything that would allow us to self-host everything easily in the future (when costs become prohibitive). I believe docker containers and k8s are the right software solution, so we just need some IaaS or PaaS to let us ship, run, and scale containers.

syself.com sounds great, but I don't know much about it. I did some searching and there are some other kubernetes orchestration services, but I don't know anything about them either.

1. Civo Kubernetes
2. Vultr Kubernetes
3. DigitalOcean Kubernetes

[SgtPooki]

I think one more thing we will need here that hasn't been fully discussed is something like Uptime Kuma to monitor a health state across all services. This should be fairly lightweight.. and could probably be piped into grafana

[BigLep]

Thanks for the comments here @SgtPooki and @Kubuxu . I won't be able to engage here more until later in the week.

@SgtPooki : I'll point you at the FilOz repo where we have kubernetes infra with AWS. I'm not saying that needs to be replicated, but it's something that would be good to be aware of.

• https://github.com/filecoin-project/filoz-infra (uses kubernetes)
• https://github.com/filecoin-project/lotus-infra (doesn't use kubernetes I don't think)

[hugomrdias]

For logs, metrics, traces, errors (sentry compatible), uptime, status page, etc all in one place its better stack. From my research it seems to best option and worth the price.

Coolify in hetzner can be a good simple solution for lightweight apps until we ramp up a kubernetes setup.

For the heavy stuff I agree with @Kubuxu. @Kubuxu did you create any doc with the kubernetes research we did ?

[SgtPooki]

Do folks have any preference for local k8s cluster testing/deployment than kind (i.e. k3d, Minikube, K3s, MicroK8s, and k0s) ? kind seems like the best option to reproduce a full, non-modified, kubernetes cluster locally, and has good CI support (helm has an action specifically for kind: https://github.com/helm/kind-action)

[SgtPooki]

Another item to consider with our infra setup: Terraform is no longer open source.. it seems like many folks are switching to OpenTofu, which should be a drop-in replacement for Terraform.

[BigLep]

2025-12-15 checkin on where we're at

@SgtPooki and I have had some conversations on this topic, particularly in light of the pressure to move the dealbot forward in the right direction.

Things I believe we all agree on

• Use kubernetes
• Use GitOps
• Infrastructure as code so infra changes can be reviewed/reverted
• Generally any "user state" we care about should be backed up (e.g., databases should be backed)
• Generally isolate services as much as possible (e.g., don’t share DBs )
• Monitoring logs: use BetterStack because
  • We have already been having success with it
  • Supports backend and Sentry logs
  • Seems like a reasonable thing that we'll want to open up without needing to give access to our core infrastructure
• Metrics for dashboarding and alerts: use Grafana
• For the apps/services we're building, it should be possible to do local development/testing without the need of 3rd party cloud tools/services

Open questions:

What k8s provider?

I know there has been discussion about https://syself.com/. That does look like a nice solution, but I'm wondering why not just the EKS solution we have in https://github.com/filecoin-project/filoz-infra for now given:

1. It's setup already
2. It has GitOps
3. It has Infra as Code
4. It's handling the full solution of dealing with certificates, DNS, and public access
5. There was success previously of deploying other applications (e.g., pdp explorer before it was extracted out).

I don't like how much stuff there is in https://github.com/filecoin-project/filoz-infra/tree/main/deployments . It feels like "a lot", but I'm wondering if that's roughly what it takes to get a full production deliverable that meets the kubernets/gitops/IaC requirements above. While I don't immediatley grok it as I'm not trained on the toolchain, I believe AI tools these days have no problem with it.

@BigLep suggestion: for the dealbot (which is the most immediate next infra project to spin up) we try plugging into the filoz-infra system, and then look at how gross/ugly the PR is. If someone feels strongly though that there is definitely a better way, I'm up for hearing it. I just don't want to get caught talking too long here.

What k8s setup for local development?

@BigLep understanding: I don't know how much this matters but @SgtPooki was using kind as part of dealbot work.

Do we manage our own DBs?

@BigLep suggestIon: right now at our small scall, I don't want to have to get educated on managing postgres databases (e.g., software updates, monitoring, resizing, setting up backups). Using a hosted solution here seems fine to me as long as we're just using vanilla postgres. Apps like the dealbot should take a postgres connection screen. Local development can use a local postgres container (or whatever else the developer creates). Production would use something like RelationalDtabaseService RDS (if we're in the current filoz-infra world abovee), rather than leveraging another service like Supabase)

Where does Coolify fit into our stack?

@BigLep understanding is that at the minimum it's useful for quick prototyping where we don't want to get bogged down in k8s, helm, etc. I see it has GitOps / IaC capabilities, but it doesn't provide k8s support. As a result I'm imaging a world where we have a shared Coolify and we use that for spinning up prototypes, but when it's ready to take something to production, we do the k8s converstion and use the "production infrastructure" discussed above.

What Grafana provider do we use?

@BigLep understang is filoz-infra currently has a Grafana instance hosted/deployed. Any reason not to keep that, or would we like to get out of the business of managing it and instead use Grafana cloud?

Where does Vercel fit in?

@BigLep understanding is that Vercel is being used for sites right now that are effectively static. I assume if we do have any server-side logic, that could happen in Coolify which would let us drop Vercel.

What IaC tooling do we use?

@BigLep : I understand Terraform's license has changed. I assume for now we'll use it for IaC and in future consider switching if/when needed (e.g., use OpenTofu).

Next Steps

Suggested next steps:

1. Per above, dealbot get started doing [Op Readiness: P0] filoz-owned infra deployment dealbot#88 using filoz-infra so we keep moving and gage just how annoying it is creating new apps/services within the filoz-infra setup.
2. Get feedback from others (e.g., particularly from @Kubuxu who has spent more time with filoz-infra) how much he's opposed to building on the filoz-infra foundation for now knowing that the main abstraction we're using in k8s, so we should be able to switch to something else with some effort in the future if needed.
3. TPMs start doing the inventory of infra we own today. Lets use https://www.notion.so/filecoindev/FilOz-Infrastructure-2cbdc41950c180f58efee5b9ad5a1203 for now to build that out.