I've built things that handle less than 1 tps and things that handles more than a few thousand without any significant memory or CPU load. All of these things have had uptime that has been unmatched with other systems that it has had to interface with all the while degrading gracefully and handling everything else in between. So lets just say I understand a thing or two about designing fault-tolerant systems that need to operate under high loads and degrade gracefully.
* Re: workload in the 70s. You are missing the point about logistics stacks than handle 1000s of transactions per second. The point is that Uber's problem is self-imposed. Stepping back and thinking about the problem a little will let them handle the same amount of work with 1/10 the hardware costs.
* Re: first version. The first version and the n-th version when properly designed requires the same set of gradual steps. If you build the first version to throw away then whose problem is it that you built it that way and need 10x the hardware to handle the workload because of shitty architecture? Again, stepping back and taking a holistic view and thinking a little bit is the trick.
* Re: extensibility. Same deal. Design your architecture properly and you can extend it as far as any business requirement forces it without spending 10x on hardware and software. How do you do this? Same as above. Thinking.
* Re: reliability. See above. Thousands of transactions a second with unmatched uptime. It is more likely the systems I interface with will go down or even for AWS to have an outage than for a properly designed system to fail.
1. Machine learning - already doing it wrong. You've failed to learn from history and instead are following fads and trends. When properly framed routing/allocation is a linear program and there are solvers than will solve such problems with millions of variables. Instead you have opted to complicate the problems with latest fads and trends that are not even suited to the problem you are solving. In essence you've made my point.
2. Consumer laptop? I'd hope the software runs on server grade hardware. Bringing up a consumer laptop as a restriction on memory is a non-sequitur.
3. Hundreds of thousands. Great. I can handle several thousand connections per second on a dinky c4.2xlarge instance with 10-20ms guarantee with a ruby stack. There are plenty of ways to optimize it further but I've never needed to. The literature is full of optimized and distributed graph search algorithms. Operationalizing any one of them wouldn't be much work. How do I know? Because I've done it before.
4. Reliable data pipelines have been a solved problem since hadoop and friends. This is a solved problem. Again making my point about bloat.
Re: one team over a year. Seems like you need better engineers or better designed systems. If you're developing software with more than 100 engineers and the boundaries between teams are so ill-defined that you need more than 10 per team then that's an organizational problem and highly inefficient way to do things. How do I know? Worked on teams that gelled and those that didn't. The determining factor was always reducing communication overhead by proper architectural design. The amount of communication overhead was almost directly correlated with software bloat and sprawl.
1. Devops: Solved problem. Chef, ansible, puppet. Pick one they're all the same.
2. Kafka is not good software. Pick something else for your event management pipeline. Heck, build it from scratch. Neither Kafka nor Storm are novel or required. Chances are you've over-engineered it if you are reaching for those and need to step back and think.
3. Simplify your call graph. There is no magic bullet here. No amount of dashboards, logs, and metrics will let you get around an ill-designed and bloated service architecture. Again you've made my point.
Same reason any other software is not good software. Chances are you don't need it and are reaching for a shiny tool. Kafka requires zookeeper and in my experience zookeeper is an operational nightmare. If you need an event bus then there are many out there that are much simpler and easier to maintain operationally with much simpler failure modes.
Don't just reach for something because it has been the most common thing posted on programming forums. The behavioral psychologists and economists consider this a well known cognitive bug.
Rabbitmq, perfectly fine message bus in pretty much all use cases. Easier to operate and maintain without any extra dependencies and much simpler failure modes. A few more: zeromq, sqs, hornetq, nats, nsq, etc. Any one of those will most certainly fulfill whatever use case you have.
The point being kafka has a very heavy operational overhead and you better understand what you are getting into and what bargain you're making for the scalability you mention.
I've used Rabbitmq, it most certainly does not fulfill the volume requirements I have.
The fact that you are comparing zeromq to Kafka is pretty good evidence that you have no idea what you are talking about, and are just tossing out names from google. I'm a little disappointed, honestly, I hoped you were aware of something I hadn't heard of.
There are two ways to solve problems in engineering. You either bring the problem closer to your existing solutions by redefining the problem or you keep the problem the same and bring your solutions closer to the problem.
Sounds like you are unwilling to redefine your problem so that it is amenable to solutions that are not kafka.
Yeah, you recommended a sockets library as an alternative to a distributed durable circular buffer. Not obviously clueful. Might as well recommend Nginx as an alternative to JavaScript.
I need to durably handle billions of events per day. No amount of redefining changes the underlying business problem.
Kafka, on the other hand, has been helping me solve that problem for years.
Let's see. I can handle a few million on a single instance and I have yet to hit any memory or CPU limits indicating I can handle 10x of what I'm currently handling. Oh and it's about 100k or more transactions per hour at peak load. Just from basic operational observation and logs. Also, have yet to see any durability issues and I've managed to do it without kafka. So pretty basic math says the entire thing can be scaled to a "few billion" transactions in a pretty straightforward way. Then again I'm more willing to redefine my problems to come up with simpler solutions.
But this discussion has devolved into personal insults at this point. We have nothing to teach each other it seems.
If nothing else, I could teach you that zeromq has nothing to do with queing or durability.
It's certainly possible that Rabbit has improved in the years since I used it, if it works for your use cases, great. But don't assume that everyone using a popular technology is doing so because of a fad or without understanding the tradeoffs.
* Re: workload in the 70s. You are missing the point about logistics stacks than handle 1000s of transactions per second. The point is that Uber's problem is self-imposed. Stepping back and thinking about the problem a little will let them handle the same amount of work with 1/10 the hardware costs.
* Re: first version. The first version and the n-th version when properly designed requires the same set of gradual steps. If you build the first version to throw away then whose problem is it that you built it that way and need 10x the hardware to handle the workload because of shitty architecture? Again, stepping back and taking a holistic view and thinking a little bit is the trick.
* Re: extensibility. Same deal. Design your architecture properly and you can extend it as far as any business requirement forces it without spending 10x on hardware and software. How do you do this? Same as above. Thinking.
* Re: reliability. See above. Thousands of transactions a second with unmatched uptime. It is more likely the systems I interface with will go down or even for AWS to have an outage than for a properly designed system to fail.
1. Machine learning - already doing it wrong. You've failed to learn from history and instead are following fads and trends. When properly framed routing/allocation is a linear program and there are solvers than will solve such problems with millions of variables. Instead you have opted to complicate the problems with latest fads and trends that are not even suited to the problem you are solving. In essence you've made my point.
2. Consumer laptop? I'd hope the software runs on server grade hardware. Bringing up a consumer laptop as a restriction on memory is a non-sequitur.
3. Hundreds of thousands. Great. I can handle several thousand connections per second on a dinky c4.2xlarge instance with 10-20ms guarantee with a ruby stack. There are plenty of ways to optimize it further but I've never needed to. The literature is full of optimized and distributed graph search algorithms. Operationalizing any one of them wouldn't be much work. How do I know? Because I've done it before.
4. Reliable data pipelines have been a solved problem since hadoop and friends. This is a solved problem. Again making my point about bloat.
Re: one team over a year. Seems like you need better engineers or better designed systems. If you're developing software with more than 100 engineers and the boundaries between teams are so ill-defined that you need more than 10 per team then that's an organizational problem and highly inefficient way to do things. How do I know? Worked on teams that gelled and those that didn't. The determining factor was always reducing communication overhead by proper architectural design. The amount of communication overhead was almost directly correlated with software bloat and sprawl.
1. Devops: Solved problem. Chef, ansible, puppet. Pick one they're all the same.
2. Kafka is not good software. Pick something else for your event management pipeline. Heck, build it from scratch. Neither Kafka nor Storm are novel or required. Chances are you've over-engineered it if you are reaching for those and need to step back and think.
3. Simplify your call graph. There is no magic bullet here. No amount of dashboards, logs, and metrics will let you get around an ill-designed and bloated service architecture. Again you've made my point.