The very definition of over-engineered. This is just event-sourcing turned into a marketing article for Kafka.
It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll need. Add in an elasticsearch cluster on the side and problem solved.
Any database can serve as a log replay, as long as you save all the versions then it's just called a query.
> The article addressed their potential issues with snapshotting
They say it would be outdated - but if you store all the versions like I said, what is getting outdated? Kafka consumers are essentially doing the same thing - it's a poll-based model that asks for more data from the current offset, no different than a SQL query with a where clause.
Unless I'm mistaken, If I were to build out a simple event log represented by a relational DB, I have bottle necks when writing to it, and have lag in terms of processing the events, and if I were also pushing those events to a queue to hydrate aggregate snapshots I would have to have client logic to deal with duplicate events or not acking processed events etc?
Intuitively, I guess kafka is "more realtime" and "more available" when compared to the home-brew event log?
EDIT: obviously those constraints in my home-brew event log are relaxed when my problem domain is amenable to things like associative operators, idempotency, inverses etc.
There's no reason not to make good use of Kafka or similar solutions. The issue is that people use it without understanding it. In this article, they say that Kafka is their system of record and their primary long-term storage. That's very silly.
You can use Kafka as the buffer/processing log before persisting to the database, but with such a small dataset it's just not necessary. It's a news publishing system, not high-frequency trading.
Well my point is that it's probably faster to get to production if I simply used Kafka _when modelling my work flow as an event processing system_ but it took them a year so I don't know now haha
A year is actually not that much for porting such a huge legacy system to a new platform. I imagine most of the work was making the interfaces from/to other platforms.
it's exact opposite the main cost is gc of dead rows in MVCC RDBMS since you are never deleting or updating rows performance will be very decent for writes.
I was about to comment that while any database can serve as a log replay and may even be feasible for these volumes, scaling it at high volume later would be extremely hard - I have experience with this in a multi-petabyte set up and its a nightmare using an RDBMS.
But, the article says "In Apache Kafka, the Monolog is implemented as a single-partition topic" - losing all the goodness Kafka provides around scaling. This setup now is no better than an RDBMS with master/slave replication.
Event sourcing is unnecessary here. What they want is a versioned history of their content with flexible schemas and arbitrary queries. Instead of using a strong fast graph database as a perfect fit, they chose to implement it poorly using Kafka which I see as much more time wasted on reinventing the wheel.
The dataset is small and all of the "very different" use-cases are just downstream apps that query a database. Why use Kafka to then materialize several different databases when a single graph database can serve all these downstream apps? Remember Kafka consumers themselves are just polling queries against a log.
A processing log is one thing but event-sourced source of truth in custom database logic in Kafka is borderline ridiculous. This project is more moving parts and less functionality while cleaning none of the existing mess. Effort would've been better spent consolidating all their systems instead (which they still have to do since these standard schemas need to be used somehow).
That doesn't prove that the NYT's money was poorly spent. Decisions are always slow and expensive, and typing is quick. Copying an existing design is almost always cheaper than making the original.
And in many cases, copying is dramatically cheaper. My favorite example is the iOS game Threes[1], which took 14 months to make and then was cloned on android in ~20 days. It was cloned so quickly that the original developers were accused of copying the cloned version!
But just because the cloners made a functionally identical product doesn't mean they did the same work as the original designers. They got to skip designing anything - which is usually the hardest and slowest part.
They claim in the article to have 100GB of text data. Let's bump that up 2 magnitudes for all text and metadata ever (outside of media files) and you can still run the entire thing on a single rack of servers and meet any performance needs.
Many industries and applications are leagues ahead in both data size and speed - this isn't an example of such.
100GB...let me go find the largest MicroSD card in my house to put that on. That would actually be a good fit, since it'd work in an rPi3 which could likely serve their data publishing needs (assuming only a few updates to articles per second..not mentioned in the article, but I'd be surprised if there's more than that given the data sources) vs what they've done.
Honestly, what kind of RPS are they talking here? Requests per minute, if that, seems like.
It doesn't really matter what the "source of truth" system is, although Kafka doesn't seem quite as mature/stable enough for that. With such little data, they can push into a nice graph database instead, run it entirely in memory and meet 10x any demand they'll ever see along with all the query types they'll need. Add in an elasticsearch cluster on the side and problem solved.
Any database can serve as a log replay, as long as you save all the versions then it's just called a query.