1PB of EBS volumes on sc1-class HDD is roughly $15k/month.
Let's say you need at least 2 copies of each byte for cross-AZ availability and durability, that's 30k per PB for JUST the drives.
These disks also need compute attached to them, coupling their scaling needs. Now you have a fleet of (expensive) VMs, regardless of how many queries end up hitting that database.
Of course, those VMs have to run pretty complex software such as PlanetScale, which has to take care of replication, backups, concurrency control, sharding, indexing, vaccuuming, and a whole bunch of other things that add operational complexity and require expertise.
That's the fixed costs of an OLTP database. A lot of that cost is derived from the need to, well, process transactions.
Thing is, they already have a data lake. So the fixed cost to store a PB of data on S3 is already paid. They mention the index is roughly 3 orders of magnitude smaller than the data set (1PB of data is roughly 1TB of index).
With a marginally less complex compute stack, I believe this becomes cost effective pretty quickly.
It was designed with large-scale environments in mind. I’m aware of several deployments managing hundreds of petabytes of data and billions of objects, which is why lakeFS does not use Git’s Merkle tree / directory tree approach.
very very fast: proportional to the count of objects modified, but not their size. Every commit represents a snapshot - an immutable listing of objects that represents the repository. reverting is essentially applying the inverse of the diff introduced by the reverted commit.
This is metadata only as the objects themselves are immutable.
Imagine an agent dropping a directory with 1m images in it. just figuring out what happened and what got dropped, restoring it one by one, etc. - doable, but ergonomics are a bit lacking.
Regarding pricing - that's indeed a great question and we don't have an answer yet. It will very likely be based on consumption and should be competitive to similar solutions.
Atomic commits are based on snapshotting done by lakeFS under the hood. Each sandbox run produces a new atomic commit to a hidden "main" branch. Updating that branch is optimistically concurrent, with lakeFS checking for conflicts - multiple writers updating the same object.
It uses lakeFS under the hood, so the unit of conflict would be a single file (object, under the hood). Resolving conflicts requires "picking" a winning side, or rerunning a conflicting job. Would you see a use case for merging changes into the same file? Interested to hear about your use case!
We're building a CAD for drug design, we often have to handle large and highly varied file formats. Protein structures, compounds, python scripts, lab notebook entries, instrumentation data, etc.
From a data structure and file ergonomics perspective, think of it as similar to Unity or UE4 for drug design. We have a huge variety of assets to manage alongside their relationships to each other, and the project files are local on the user's machine (with a collaboration / sync over the network between scientists working on the same project, hence where something like this would come in for us).
Many of those files are fine with a winning side strategy, but some of them might not be that clean. Take a protein structure defined by an `mmcif` file for example, if we clean the file by removing hydrogen atoms and another scientist repairs a side chain on that same file then we'd need a way to reconcile those differences.
On the agent side, our agents will generate small python scripts that manipulate the proteins, then cache and re-use those scripts as tools when possible. So preserving those scripts alongside the mutated asset and conversation history is something we've been working on.
It provides a filesystem abstraction, which agents are really good at interacting with. Because it's just a POSIX filesystem - you can put a sqlite database directly on it and get those same transactional capabilities for that too.
it should absolutely scale to that. The filesystem is backed by lakeFS, where every sandbox automatically branches out, and mounts that branch. so you get isolation from lakeFS and the scale of an underlying object store (S3, in Tilde).
If that database is stored on the transactional filesystem available to the sandboxes, yes! Instead of backing up, it utilizes an efficient snapshot mechanism (lakeFS under the hood).
Part of the appeal (subjective, I know) of versioning is stuff like human-in-the-loop approvals. Think of a pull request: a change is requested by an agent, a human approves, changes get merged atomically. Even if other changes were applied since creation.
1PB of EBS volumes on sc1-class HDD is roughly $15k/month. Let's say you need at least 2 copies of each byte for cross-AZ availability and durability, that's 30k per PB for JUST the drives.
These disks also need compute attached to them, coupling their scaling needs. Now you have a fleet of (expensive) VMs, regardless of how many queries end up hitting that database.
Of course, those VMs have to run pretty complex software such as PlanetScale, which has to take care of replication, backups, concurrency control, sharding, indexing, vaccuuming, and a whole bunch of other things that add operational complexity and require expertise. That's the fixed costs of an OLTP database. A lot of that cost is derived from the need to, well, process transactions.
Thing is, they already have a data lake. So the fixed cost to store a PB of data on S3 is already paid. They mention the index is roughly 3 orders of magnitude smaller than the data set (1PB of data is roughly 1TB of index). With a marginally less complex compute stack, I believe this becomes cost effective pretty quickly.