Robust Delete and Time To Live (TTL) Functionality Now Available in FeatureBase
The idea of “deletes” in a database can be controversial due to the concern that data could be “damaged” or irretrievably manipulated.
When it comes to real-time databases powering production workloads, the volume of data can overwhelm the storage and memory available in the database, resulting in a myriad of problems.
Because of this, a real-time, production-grade database needs to have some form of mitigation. In other words, a way to delete data or views of data that are no longer necessary in order to free up memory. If this doesn’t exist, hardware requirements will spin out of control, creating an untenable TCO for a business to maintain.
Additionally, as data privacy regulations gain more traction worldwide, the ability to delete data is becoming a requirement to engage in international business (e.g., GDPR’s “Right to be Forgotten”).
To prevent unbounded storage and memory growth, our engineering team went to work building robust Time to Live (TTL) and Delete() functionality in FeatureBase. We have customers who ingest huge amounts of data each day, so it’s not feasible or efficient to keep it all—especially when that data becomes irrelevant to their use case after a certain amount of time. Using one or both of these tools can help pare down overwhelming data volumes.
Deletes in FeatureBase
Previously, FeatureBase had some delete capabilities:
- “Clear” and “ClearRow” operations allowed users to set bits to zero
- Delete operation that allowed setting all the bits for a particular set of records to zero but did not delete from key translation stores
- The ability to delete whole fields and tables
With our latest release, deletion in FeatureBase now includes:
- Delete operation that can delete from record key translation
- The capability to reuse a given record’s “slot” once it has been deleted
With this new functionality, customers can add a “last updated” timestamp field to each record and periodically issue a Delete() query. This functionality will remove all records that haven’t been updated within the past “X” number of days.
Time to Live (TTL) in FeatureBase
Along with our new delete functionality, we also implemented a Time to Live functionality (TTL). Using TTL, FeatureBase can expire database records based on user-created settings and reclaim unused space by deleting views that are no longer necessary; this is expressed as the amount of time data is allowed to live in the database.
With this capability, customers can now set time fields to expire views containing bits older than the specified time window.
What Deletes and TTL Mean for Molecula Customers
At Molecula, we’re laser-focused on meeting the requirements of customers who need real-time access to massive-scale datasets. Upgrading our delete functionality and adding TTL enables our customers to experience the power of FeatureBase’s feature-oriented format without worrying if resources are being wasted on irrelevant/unnecessary data.
These updates empower our customers to operate on mission-critical, real-time data without the need for thousands upon thousands of servers. Literally. In one case, a customer downsized from a multi-hundred node cluster to just a handful of FeatureBase servers.
If you’re interested in learning more, keep an eye out for our upcoming “How To Use TTL” blog or contact our team to request a demo. As always, kudos to our Product and Engineering teams for their hard work bringing these capabilities to life in record time!