Show HN: The Future of Data APIs Hello everyone, I'm Matteo, and, over the last few months, I have been working with my co-founder and other folks from Goldman Sachs, Netflix, Palantir, and DBS Bank to simplify building data APIs. I have personally faced this problem myself multiple times, but, the inspiration to create a company out of it really came from this Netflix article (https://ift.tt/r8LnHfR). You know the story: you have tons of data locked in your data platform and RDBMS and suddenly, a PM asks to integrate this data with your customer-facing app. Obviously, all in real-time. And the pain begins! You have to set up infrastructure to move and process the data in real-time (Kafka, Spark, Flink), provision a solid caching/serving layer, build APIs on top and, only at the end of all this, you can start integrating data with your mobile or web app! As if all this is not enough, because you are now serving data to customers, you have to put in place all the monitoring and recovery tools, just in case something goes wrong. There must be an easier way !!!!! That is what drove us to build Dozer. Dozer is a simple open-source Data APIs backend that allows you to source data in real-time from databases, data warehouses, files, etc., process it using SQL, store all the results in a caching layer, and automatically provide gRPC and REST APIs. Everything with just a bunch of SQL and YAML files. In Dozer everything happens in real-time: we subscribe to CDC sources (i.e. Postgres CDC, Snowflake table streams, etc.), process all events using our Reactive SQL engine, and store the results in the cache. The advantage is that data in the serving layer is always pre-aggregated, and fresh, which helps us to guarantee constant low latency. We are at a very early stage, but Dozer can already be downloaded from our GitHub repo (https://ift.tt/Z18ibAV). We have taken the decision to build it entirely in Rust, which gives us the ridiculous performance and the beauty of a self-contained binary. We are now working on several features like cloud deployment, blue/green deployment of caches, data actions (aka real-time triggers in Typescript/Python), a nice UI, and many others. Please try it out and let us know your feedback. We have set up a samples-repository (https://ift.tt/b0Lwvtr) for testing it out and a Discord channel (https://ift.tt/KmbRs5v) in case you need help or would like to contribute ideas! Thanks Matteo April 5, 2023 at 11:16PM
Comments
Post a Comment