The trick is to have your gateway handle authn, and then proxy authz data upstream so those services can decide how to handle it without needing to make a second call to the identity service.
You probably want to have a UI for account creation and password resets, right? There's a frontend that has to talk directly to identity service.
You may want to bill based on # of active users - well that's interactive with the identity service (you can do this without billing calling the identity services' API, but the alternatives are just other common dependencies.
You may want a tool for the support team to search identity service to find a user or their account status.
If you have a sharing feature, you may want that to verify you are sharing with an account that exists.
The way I've set these things up, nothing talks directly to the identity service. The ID service is a backend behind your gateway like any other service and any UI would have to have the request proxied through the gateway to reach it. Now, you can carve out certain rules (if you control the gateway) where requests headed to /users/* don't require the same authN steps other requests do because it's already headed to the ID server. Internal UIs may or may not work the same, that's really up to you - they won't likely be super high scale. Often the support teams won't even be querying the real DB, but instead a view or copy so they can't affect real user data. A share code for users A->B would just be a request from the UI to the ID server via the gateway, authenticated as User A, and responding with the code for B if possible. Or, I've do it where you could have special logic in the gateway to query 2 servers and combine the responses. No need for services to make requests sideways. Hope that makes sense.
The trick I've used is the N1 (gateway) service handles all AuthN and proxies that information to the upstream services to allow them to handle AuthZ. N+ services only accept requests signed by N1 - the original authentication info is removed.
Treating N4 as a service is fair. I think the article was leaning more toward that idea of N4 being a database, which is a legit bad idea with microservices (if fact defeating the point entirely). My takeaway is that if you're going to have a service that many other services depend on, you can do it but you need to be highly away of that brittleness. Your N4 service needs to be bulletproof. Netflix ran into this exact issue with their distributed cache.
Ah yes, and every consumer should just do this in a while (true) loop as producers write to it. Very efficient and simple with no possibility of lock contention or hot spots. Genius, really.
I've implemented a distributed worker system on top of this paradigm.
I used ZMQ to connect nodes and the worker nodes would connect to an indexer/coordinator node that effectively did a `SELECT FROM ORDER BY ASC`.
It's easier than you may think and the bits here ended up with probably < 1000 SLOC all told.
- Coordinator node ingests from a SQL table
- There is a discriminator key for each row in the table for ordering by stacking into an in-memory list-of-lists
- Worker nodes are started with _n_ threads
- Each thread sends a "ready" message to the coordinator and coordinator replies with a "work" message
- On each cycle, the coordinator advances the pointer on the list, locks the list, and marks the first item in the child list as "pending"
- When worker thread finishes, it sends a "completed" message to the coordinator and coordinator replies with another "work" message
- Coordinator unlocks the list the work item originated from and dequeues the finished item.
- When it reaches the end of the list, it cycles to the beginning of the list and starts over, skipping over any child lists marked as locked (has a pending work item)
Effectively a distributed event loop with the events queued up via a simple SQL query.
Dead simple design, extremely robust, very high throughput, very easy to scale workers both horizontally (more nodes) and vertically (more threads). ZMQ made it easy to connect the remote threads to the centralized coordinator. It was effectively "self balancing" because the workers would only re-queue their thread once it finished work. Very easy to manage, but did not have hot failovers since we kept the materialized, "2D" work queue in memory. Though very rarely did we have issues with this.
Yeah, but that's like doing actual engineering. Instead you should just point to Kafka and say that it's going to make your horrible architecture scale magically. That's how the pros do it.
Kafka isn't magic, but it's close. If a single-node solution like an SQL database can handle your load then why shouldn't you stick with SQL? Kafka is not for you. Kafka is for workloads that would DDoS Postgres.
Kafka is really not intended to improve on this. Instead, it's intended for very high-volume ETL processing, where a classical message queue delivering records would spend too much time on locking. Kafka is hot-rodding the message queue design and removing guard rails to get more messages thru faster.
Generally I say, "Message queues are for tasks, Kafka is for data." But in the latter case, if your data volume is not huge, a message queue for async ETL will do just fine and give better guarantees as FIFO goes.
In essence, Kafka is a very specialized version of much more general-purpose message queues, which should be your default starting point. It's similar to replacing a SQL RDBMS with some kind of special NoSQL system - if you need it, okay, but otherwise the general-purpose default is usually the better option.
Of course this is not the same as Kafka, but the comment I'm replying to:
> Ah yes, and every consumer should just do this in a while (true) loop as producers write to it. Very efficient and simple with no possibility of lock contention or hot spots. Genius, really.
Seemed to imply that it's not possible to build a high performance pub/sub system using a simple SQL select. I do not think that is true and it is in fact fairly easy to build a high performance pub/sub system with a simple SQL select. Clearly, this design as proposed is not the same as Kafka.
No, I implied that implementing pub/sub with just a select statement is silly because it is. Your implementation accounts for the downfalls of this approach with smart design using a message queue and intelligent locking semantics. Parent of my comment was glib and included none of this.
It's one of my favorite patterns, because it's the highest-impact, lowest-hanging fruit to fix in many systems that have hit serious scaling bottlenecks.
pull vs push. Plus if you start storing the last timestamp so you only select the delta and if you start sharding your db and dealing with complexities of having different time on different tables/replication issues it quickly becomes evident that Kafka is better in this regard.
But yeah, for a lot of implementations you don't need streaming. But for pull based apps you design your architecture differently, some things are a lot easier than it is with DB, some things are harder.
A Kafka consumer does a lot of work coordinating distributed clients in a group, managing the current offset, balancing the readers across partitions, etc which is native broker functionality. Saying you can replace it all with a simple JDBC client or something isn't true (if you need that stuff!)
What you're doing is fine for a homelab, or learning. But barring any very specific reason other than just not liking Kafka, its bad. The second that pattern needs to be fanned out to support even 50+ producers/consumers, the overhead and complexity needed to manage already-solved problems becomes a very bad design choice.
Kafka already solves this problem and gives me message durability, near infinite scale out, sharding, delivery guarantees, etc out of the box. I do not care to develop, reshard databases or production-alize this myself.
Some people don't and won't need 50+ producers/consumers for a long while, if ever. Rewriting the code at that point may be less costly than operating Kafka in the interim. Kafka is also has a higher potential for failure than sqlite.
Ofc, and not everybody needs or cares for all the features Kafka has. Then use another known and tested messaging system. Use NATS or ZMQ. Or any cloud native pubsub system
My main point is, I have zero interest in creating novel solutions to a solved problem. It just artificially increases the complexity of my work and the learning curve for contributors.
Have a table level seqno as monotonically increasing number stamped for every mutation. When a subscriber connects it asks for rows > Subscriber's seqno-last-handled.
If you haven't been following space updates closely, the US is _already_ in a race with China, especially in regards to the Artemis (moon) missions. That being said it's mostly being used as an excuse to keep SLS alive and prop up the legacy space contractors... It's hard to lose a contest you won 60 years prior...
> It's hard to lose a contest you won 60 years prior...
If you pick a random person off the street and ask them who discovered the Americas will they answer 1. Leif Erikson, 2. Indigenous peoples or 3. Christopher Columbus? If you ask people who invented the smartphone will they say Apple or some other company?
It’s absolutely possible to lose a race you had previously won.
Or indeed, ask them who won the space race, because by most measures, that was the Soviets too.
Soviets achieved:
- First artificial Orbit ( Sputnik )
- First animal to orbit ( Laika )
- First Man to orbit ( Yuri Gagarin )
- First Woman to orbit (Valentina Tereshkova )
- First EVA ( Alexei Leonov )
- First moon landing ( Luna 9 )
- First landing on another planet ( Venera 8 )
Many of these years before the USA achieved the equivalent. The first female US astronaut wasn't until the mid 1980's.
The Americans were at one point beat so bad that they invented their own game that only they were playing.
Yes, that spurred their entire economy and the boosted scientific investment paved the way for the decades of dominance since, and that should be rightly celebrated, but the idea that the USA "Won the space race" because of the moon landing is Hollywood nonsense.
They are still jerking on these 50+ years old achievements, without having new ones. "Space race" didn't stop after Venera landing, and soviets/russians are thing of the past now. Aside of useless ISS trips, they have no relevance in space anymore.
> "Won the space race" because of the moon landing is Hollywood nonsense.
"Won the space race" because they were first at the very beginning is a nonsense too. Following this logic, China won rocket race because they invented first rockets centuries ago.
What use is the stuff that went on 60 years in the past when most people involved with it are dead and a lot of the knowledge has been lost in the meantime?
I remember picking through an aerospace scrapyard in North Hollywood a decade ago with extremely-talented launch engineers (and entrepreneurs). The aim was to look at parts, measure them and figure out why they were built like they were. We looked, a little, at stuff like nozzles. But mostly we focussed on bolts, joiners, turbine blades and the like.
It's written to mimic that style but without meaning that the work has been done for them, just that there is new work to be done, making it an odd perhaps unconscious reference
If you do use terraform, for the love of god do NOT use Terraform Cloud. Up there with Github in the list of least reliable cloud vendors. I always have a "break glass" method of deploying from my work machine for that very reason.
Off topic, and I don't want to knock the presenter here, but if you're ever going to give a public talk or presentation at work _please_ review the Death By Powerpoint slide deck[0] first.
Thanks for the link. I went through it but I'm not sure what it's telling me to change. Can you elaborate? If it's "be more engaging", i think is unfortunately going to be hard to improve because this is just how i am.
I appreciate your frank oppenness to feedback! I'd like to give you some, but first I want to call out that (1) I'm certainly no expert and (2) you may have been voluntold to make a powerpoint on material that isn't well-fit for powerpoint.
My feedback:
- I think you could improve by being more excited / exciting (the Death By Powerpoint slide deck says "passionate"). Personally, I try to mitigate this problem by drinking coffee shortly before I'm expected to be engaging. It is a drug, and for me it works.
- You could also draw the main impactful points out of the background info. The presentation you give here has a lot of deep technical detail, which is certainly of interest for the correct audience, but I think generally fits poorly in a powerpoint presentation. I think you'd generally want to focus on more high-level performance characteristics, or particularly interesting details. Perhaps a 3-4 page document would be better for this sort of deep technical material. Or maybe the level of detail is actually perfect, and I'm just not the correct audience.
I really love your response here. You seem like a great person, I think your coworkers are lucky to have you.
The presentation was mainly for the audience that was present at the conference, which was mainly people who have used jj for a relatively long time, so I think part of the problem is just a mismatch between audiences there and on YouTube.
I agree about being more exciting. Not sure how to improve that :P I drink too much coffee all the time that I don't notice any difference. Harder drugs, perhaps :)
Which is fair as is not expecting everyone to be able to deliver a barnstorming talk. At least not without training.
I'd also flag that the audio is not good which isn't your fault. The trend towards conferences just pointing a camera at a lectern and throwing the footage on YouTube is unfortunate especially when it replaces circulating the slides as opposed to augmenting them. As a form of archive for attendees to review it works but not as a distribution channel.
I think GP is suggesting using less words on a slide, having visual aids and often, thinking of 2-3 ideas that someone would remember from your presentation. To the first point, if you take more than 1 min on a single slide, you need to split it into more slides.
Also, present things that you are passionate about if you want to make people care about your presentation.
Thanks for presenting. I was a fig lover so this area is interesting to me.
This is a really hard topic since it's hard to understand the significance of the jj contributions without explaining a ton of internal details like Piper, CitC, TAP, blaze, etc. I think you struck a good balance here.
I don't think you need to "be more engaging" or be more passionate, and when people say that, it indicates they aren't engaged, but it's not a useful suggestion of how to improve.
I think the main issue was that on just about every slide, I found myself wanting to read ahead on the slides instead of waiting to listen to you talk. There are a couple factors to this:
1. Impatience. I want to ingest information faster. Believe it or not, the "talk slowly" advice is usually not good for a technical talk with decent acoustics. E.g. listen to a talk by Bryan Cantrill and notice how fast he talks while still being clear. Rehearsal helps, but if you simplify your slides, you can also make them easier to perform. Listen to a random slide in your video and notice how much time is filled with silence or filler words.
2. Scanning the slide because I'm unsynchronized with the speaker. E.g. if everything you say isn't clearly correlated with the content of the slide in a linear way, then I have to scan the slide to attempt to "resychronize" and figure out what bullet you are on. Sometimes presenters don't want to read the bullets verbatim, which makes it difficult to match up the spoken language with the written language, whereas if you said the first word in a bullet, I could quickly seek to that line. Skipping bullets also causes this, especially at the end of a slide, since then I want to try to read all the bullets on a slide ahead of you in case you decide to skip them in the future.
Fewer bullets per slide and fewer words per bullet help with both of these. Aggressively cut or summarize material so you don't have to skip over anything (e.g. instead of listing 6 jj piper commands and only talking about 2, list just 2 and write the number of total commands e.g. "we added 22 subcommands including mail and submit"). Put extra material (if any) in an appendix after the end if you think you might have extra time or want to use it for Q&A. Rehearse until you can belt out what you want to say on each slide without fillers or pauses (and record yourself so you can tell how much you are doing this). When I'm sharing my screen, I actually like to point my mouse cursor at the bullet I'm on so nobody has to guess where we are, but sometimes that's not available.
Your first slides essentially teach the audience how to consume your talk. E.g. a good pattern is if (1) all your bullets are short (2) you read every bullet verbatim, elaborate on that bullet for a bit, and then repeat on the next bullet. If you stick with a pattern like that, then your audience knows how to follow along. Other patterns work too.
I'm sorry this wasn't shorter and I hope it's helpful.
Again, amazing to be seeking feedback (or open to it). I'm at minute 13 (code review management / bookmarks, etc.), and will give my feedback from there:
"SCAR", which I learned as "Situation, Consequences, Actions, Results" but the above essay will let you go deeper.
Bad: "Two people stood up there, said a few words, and then everybody had cake and went home"
Good: "The situation was... it was a beautiful venue, everybody was dressed nicely, there was music, a slow walk. What these two people were doing was going to bond them together for the rest of their life (or until they had an ugly divorce, whichever came first). They gave a great speech that they'd been practicing all morning in the mirror, gave each other a kiss and a ring to seal the deal, then everyone had cake, danced, and went home tired but happy!"
...this is obviously (hopefully!) exaggerated, and there's a whole bunch more fluffery in the 2nd than the first but imagine this:
1) The situation is that google on perforce or git was dog slow and blah blah not designed for the scale of our monorepo, etc.
2) The consequences was reduced development velocity and increased errors in prod b/c people were trying to bypass CI/CD steps b/c everything was so slow
3) We introduced CitC / Cloud Commits, and => {your technical brilliance described here...}
4) ...and the result was butterflies and rainbows, and here's a graph that shows the production incident rates going down and the changelist rates going up after we GA'd `jj` on linux.
5) The End!
As it stands, much of the presentation is #3 but you're not even giving yourself the credit of bragging about how you discovered the problem (challenge) and the cleverness that went in to inventing your solution.
re: bookmarks:
"We often had people mailing around patches -or- Sometimes PR's were tough to review due to ... -or- I've always wanted an easy way to ..."
=> "Without good bookmark support [adoption would suffer|it would be seen as a step back|collaboration wasn't instant|...]"
=> "So I built bookmarks for `jj`, they work like ..., and solves most of the problem"
=> "It's [better|faster|cheaper|quicker|more fun] than [GitHub|Our Old System|getting yelled at by Linus Torvalds|...etc..."
=> "The IMPACT of bookmarks is: ..." => "It IMPACTED the technical bits here: ..." => "The USER IMPACT is: ..." => "...and finally the business gets: ..." => "What a powerful impact!"
I'd _LOVE_ to see you run a breakdown of any kindof arbitrary slide in the deck and post a deconstruction in this format as kindof a practice/workshop.
It's very OK if (at first) it's pretty mechanical! It's just super-helpful to basically "disassemble" what you're trying to talk about in this mechanical way, and then you can take the proper bits and put them back together.
If you just string a bunch of technical details or technical choices together, you're missing the whole "compelling story" that exists. Even if you just "set the stage" with a single "Situation" slide at the beginning, and a single "Results/Impact" slide at the end... each interior "loop iteration" can be easily set up with a short "Challenge/Consequence" & "Action/Detail/Choice"
"Git + Monorepo was yuck!" => [ "Slow FS" + "CitC" ] => [ "Big Checkouts" + "VFS" ] => [ "Branches?" + "Chose `cp $FILES branches/*` ; Feedback?" ] => "JJ has been well accepted and has a bright future, inside and outside of el-goog!"
My target audience were the people at the conference, who are mostly long-term jj users. The topic was "Jujutsu at Google: Architecture and future plans", i.e. explaining how it works and what our plans are at Google. I'm mentioning this because I think what you're suggesting is for a different presentation, more explaining what the advantages of Jujutsu at Google are rather than how it works. My goal was not to sell the solution to this audience because they're presumably already sold on it.
Sorry if this was obvious to you and you're saying that part of my problem was that I should have chosen a different topic and/or target audience (focusing more on potential new users, perhaps).
Not at all! So for example at ~20-25m, you're explaining "the revset engine"
Just basically flip the first two bullet points:
Situation: "Our repo shape can assume: ..."
Consequences: "Non-mainline graphs fit in memory ..."
Actions: "Tada! Revset engine!"
Result: "Monotonic commit numbers"
Impact: "b/c we can sort ranges => unlocks filtering, client-side processing, etc"
(please excuse my poor understanding of the DETAILS of `jj-at-goog` internals, so I'm butchering some of the details of impact, etc).
...another comment mentioned "death by powerpoint". You can still do/make your slides as you're currently doing it, but when you're "done", just move all the text down into the "speakers notes", and focus on providing some illustrations or comparisons for the visual portion.
On the revset, a bunch of it could be helped by "illustrating the impact" or "compare and contrast"
| RevSet | Git-on-ext3 |
+----------------+----------------+
| Indexed | FS-Traversal |
| Working Memory | Disk Cache |
| Strong Server | Local Limits |
| 456 > 123 | $RAND != $RAND |
...etc...
...and in your speakers notes, you can record some of the technical details you want to talk through.
You're so deep in the weeds it's like asking a fish what they think about water (and to clarify: I'm absolutely not meaning this in any sort of insulting way)... it's just that when you're communicating to others, you may have to end up moving "back and forth" or "up and down" ... illustrated: https://www.youtube.com/watch?v=3amL4O8BSAg ... stretching things out so you can see the individual threads then putting them back, or giving them a twist and blending in a new topic.
Because of your intimate familiarity with the inner workings, sometimes in your talk you're describing an inner working (ie: there's 206 bones in the human body!) but there's no context about what makes that better, or worse, or how they're conceptually connected with each other.
On the grand scale, what is the IMPACT you want your talk to have? To put words into your mouth: "We're about to go GA at google, which is a big deal."
How do you get there? Where did you start from? To put words into your mouth: "JJ is solving google-scale problems, and there's _demand_ and _support_ for bringing it about." [that's your situation]
Consequences? Actions? => now you have set the scene for your very same technical deep dives on JJ.
1) Google has big problems, JJ is solving lots of them!
2) The consequences of (jj-solving-them) or (suffering-further-with-git) are: ...
3a) The actions that have helped jj forge ahead (better than git) are: ...
3b) The actions that jj reduces suffering compared to git are: ...
4) ...as a result, we're about to go GA
5) The impact is increased community trust, improved internal development capabilities, etc...
...again: it's _mostly_ about rearranging things, and providing a little bit more context so that what you _are_ presenting hits your audience better. (3a and 3b are kindof what your current presentation was all about... just set it up a little more on how those interact with the larger picture)
Situation: You've done a great technical thing, and are being asked to present more and be an advocate for it (and yourself)
Consequences: ...if people don't understand JJ, there will be less adoption, or people that would be well-served by it won't use it, or won't contribute
Action: ...you're taking feedback on presentation skills, will continue your systematic learning and iterative improvement approach to this tech-adjacent skill
Result: Butterflies! Rainbows! 150% increase in JJ contributors and 5000% increase in JJ adoption... world peace imminent! :-P
<3 ... best of luck, and free free to reach out if you'd like to continue the conversation!
Not parent but I think the link is pretty clear about what they want to say. I haven't got the time to actually watch the full video, but I could tell this is not a great presentation just by randomly clicking at random timestamps. And almost any presentation from Google (technical or not) I watched on YouTube is better than this.
If you really want me to explain it for you, and just one issue:
There are way too many words on your slides.
P.S. You should click that link and go through it again. If you still don't get it, try once more.
If you don't care about the material, why on earth would anybody sit around for 15 minutes/30 minutes/an hour listening to you talk about the material. The sole reason for a presentation over a technical reference buried somewhere is because the presenter wants the audience to care about the thing they're presenting. If that isn't reflected in the presentation, it's not a worthwhile presentation.
But this was a technical presentation at a conference dedicated to the technology in question. A person stumbling across this on HN does not magically make them part of the target audience.
A technical presentation is a still a presentation, as opposed to a reference document. If you want to give someone a block of technical information, you do so in a reference document. You talk in front of a room full of people in order to convince them that this matters enough to bother.
Sure, but I don’t think this is relevant to the comment thread we’re in, which started off by sharing generic advice that mostly applies to TED-style motivational talks or “keynotes” at large conferences etc.
Ugh, I'd prefer people keep passion for the bedroom and just convey information straightforwardly without trying to "sell" it to me.
Adding a false excitement signal to the information is a hindrance to me as a viewer. If you want Tony Robbins then go and see him. If you want an overview of the new product architecture lets keep calm and get on with it.
I disagree, but it doesn’t mean you’re wrong - we might just be looking for different things.
My view is that if you’re going to talk like a bored robot, then I need a transcript or a paper which is much shorter than a talk. I distinctly remember not going to classes at university because the teachers wouldn’t give me anything beyond reading a book aloud. I just read the book at home.
Now, if you’re able to indicate what’s important and not important, what’s interesting and what isn’t , and why you like it , I’ll be delighted to watch your talk.
I think it’s very difficult to be interested in something new if the presenter doesn’t seem to care about it. This is different from something written , where I expect something dry anyway.
The irony the deck presents 3 very different presenters to show that you don't need to be Tony Robbins to be passionate.
If you can't be passionate, don't make it a presentation. Passionate doesn't need to mean over-the-top, but it means speaking on the topic effectively and in a way that reflects believing there's a reason for these people to sit there and listen to you.
No one is forcing people to do presentations, but once you put people in a situation where they're expected to give you their undivided attention for a block of time, those are very reasonable table stakes.
There's always blog posts, articles, mailing lists, HN posts, etc. if you don't believe there's a strong need for people to set aside time consume your information in the form of a presentation.
"passion" here means "be excited about the subject", which usually (neurospicy people exluced naturally) automatically brings up feelings from the speaker in a "this is cool and I want to share it" kind of way.
If I want to listen to someone read a slide deck word for word in a monotone voice, I can have an AI do it. I'd much rather read a blog post at that point though.
reply