DataFusion - The Database Building Toolkit (with Andrew Lamb)

Developer Voices

#Tech #News #Tech News #Kris Jenkins #Programming #Software Development #Software #Software Archtecture #Programming Language #Database

6:24

Grumpy helps a reader choose the best hydrangea for their hometown. Plus, Grumpy’s gripe of the week. You can find us online at southernliving.com/askgrumpy Ask Grumpy Credits: Steve Bender aka The Grumpy Gardener - Host Nellah McGough - Co-Host Krissy Tiglias - GM, Southern Living Lottie Leymarie - Executive Producer Michael Onufrak - Audio Engineer/Producer Isaac Nunn - Recording Tech Learn more about your ad choices. Visit podcastchoices.com/adchoices…

23 days ago 1:32:10

MP3•Episode home

Joining me to explain it all is Andrew Lamb, one of DataFusion’s core contributors, and he’s going to take us through the whole stack, how it’s built and how you could use it. Along the way we cover everything from who’s building interesting new databases and how you manage a large, open-source Rust project.

–

DataFusion Homepage: https://datafusion.apache.org/

DataFusion on Github: https://github.com/apache/datafusion

DataFusion Architecture (with diagrams!): https://youtu.be/NVKujPxwSBA?si=tw9ACxlbdpBuVsnv&t=1045

Datalog: https://docs.racket-lang.org/datalog/

Tokio: https://tokio.rs/

Andrew’s Homepage: http://andrew.nerdnetworks.org/

Andrew’s Blog Post about Tokio: https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/

Velox: https://velox-lib.io/

Arroyo: https://www.arroyo.dev/

Synnada: https://www.synnada.ai/

LanceDB: https://lancedb.com/

SDF+DBT: https://docs.sdf.com/integrations/dbt/integrating

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

88 episodes

DataFusion - The Database Building Toolkit (with Andrew Lamb)

Developer Voices

19 subscribers

published 23 days ago

MP3•Episode home

–

DataFusion Homepage: https://datafusion.apache.org/

DataFusion on Github: https://github.com/apache/datafusion

DataFusion Architecture (with diagrams!): https://youtu.be/NVKujPxwSBA?si=tw9ACxlbdpBuVsnv&t=1045

Datalog: https://docs.racket-lang.org/datalog/

Tokio: https://tokio.rs/

Andrew’s Homepage: http://andrew.nerdnetworks.org/

Andrew’s Blog Post about Tokio: https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/

Velox: https://velox-lib.io/

Arroyo: https://www.arroyo.dev/

Synnada: https://www.synnada.ai/

LanceDB: https://lancedb.com/

SDF+DBT: https://docs.sdf.com/integrations/dbt/integrating

Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices

Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join

Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social

Kris on Mastodon: http://mastodon.social/@krisajenkins

Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/

88 episodes

#Tech #News #Tech News #Kris Jenkins #Programming #Software Development #Software #Software Archtecture #Programming Language #Database

All episodes

Developer Voices

1
The State & Future of Apache Kafka (with Anatoly Zelenin) 1:12:22

10 days ago1:12:22

1:12:22

I’m joined this week by one of the authors of Apache Kafka In Action, to take a look at the state of Kafka, event systems & stream-processing technology. It’s an approach (and a whole market) that’s had at least a decade to mature, so how has it done? What does Kafka offer to developers and businesses, and which parts do they actually care about? What have streaming data systems promised and what have they actually delivered? What’s still left to build? – Apache Kafka in Action: https://www.manning.com/books/apache-kafka-in-action Pat Helland, Data on the Inside vs Data on the Outside: https://queue.acm.org/detail.cfm?id=3415014 Out of the Tar Pit: https://curtclifton.net/papers/MoseleyMarks06a.pdf Martin Kleppmann, Turning the Database Inside-Out: https://martin.kleppmann.com/2015/11/05/database-inside-out-at-oredev.html Data Mesh by Zhamak Dehghani: https://www.amazon.co.uk/Data-Mesh-Delivering-Data-Driven-Value/dp/1492092398 Quix Streams: https://github.com/quixio/quix-streams XTDB: https://xtdb.com/ Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Anatoly’s Website: https://zelenin.de/ Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ Kris on Twitter: https://twitter.com/krisajenkins…

Developer Voices

1
DataFusion - The Database Building Toolkit (with Andrew Lamb) 1:32:10

23 days ago1:32:10

1:32:10

Building a database is a serious undertaking. There are just so many parts that you have to implement before you even get to a decent prototype, and so many hours of work before you could begin working on the ideas that would make your database unique. Apache DataFusion is a project that hopes to change all that, but building an extensible, composable toolkit of database pieces, which could let you build a viable database extremely quickly, and then innovate from that starting point. And even if you’re not building a database, it’s a fascinating project to explain how databases are built. Joining me to explain it all is Andrew Lamb, one of DataFusion’s core contributors, and he’s going to take us through the whole stack, how it’s built and how you could use it. Along the way we cover everything from who’s building interesting new databases and how you manage a large, open-source Rust project. – DataFusion Homepage: https://datafusion.apache.org/ DataFusion on Github: https://github.com/apache/datafusion DataFusion Architecture (with diagrams!): https://youtu.be/NVKujPxwSBA?si=tw9ACxlbdpBuVsnv&t=1045 Datalog: https://docs.racket-lang.org/datalog/ Tokio: https://tokio.rs/ Andrew’s Homepage: http://andrew.nerdnetworks.org/ Andrew’s Blog Post about Tokio: https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/ Velox: https://velox-lib.io/ Arroyo: https://www.arroyo.dev/ Synnada: https://www.synnada.ai/ LanceDB: https://lancedb.com/ SDF+DBT: https://docs.sdf.com/integrations/dbt/integrating Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
Jupyter's Architecture Unpacked (with Afshin Darian & Sylvain Corlay) 1:29:11

5 weeks ago1:29:11

1:29:11

Jupyter’s become an incredibly popular programming and data science tool, but how does it actually work? How have they built an interactive language execution engine? And if we understand the architecture, what else could it be used for? Joining me to look inside the Jupyter toolbox are Afshin Darian and Sylvain Corlay, two of Jupyters long-standing contributors and project-steerers. They’ve going to take us on a journey that starts with today’s userbase, goes through the execution protocol and ends with a look at what Jupyter will be in the future - an ambitious framework for interactive, collaborative applications and more. – Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Jupyter Homepage: https://jupyter.org/ Jupyter Xeus: https://github.com/jupyter-xeus/xeus Jupyter AI: https://github.com/jupyterlab/jupyter-ai Jupyter CAD: https://github.com/jupytercad/JupyterCAD Jupyter GIS: https://github.com/geojupyter/jupytergis/ Jupyter GIS Announcement: https://blog.jupyter.org/real-time-collaboration-and-collaborative-editing-for-gis-workflows-with-jupyter-and-qgis-d25dbe2832a6 QGIS: https://qgis.org/ ZeroMQ: https://zeromq.org/ Sylvain on LinkedIn: https://www.linkedin.com/in/sylvaincorlay Darian on LinkedIn: https://www.linkedin.com/in/afshindarian Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
Nix, The Build-Everything Language (with Julian Arni) 1:20:36

7 weeks ago1:20:36

1:20:36

Ever since we invented makefiles, the programming world has been wrestling with the problem of building software stacks reliably. This week we’re going to look at one of the most ambitious solutions available - Nix. Nix tries to do everything from invoking your compiler to installing your language, and even providing your operating system. But how does it work in theory, and how well does it work in practice? Joining me to discuss is Julian Arni, a Nix-enthusiast and creator of a build/test/deploy service called Garnix. Nix has been one of my go-to tools for years - I hope it’ll find its way into your stack. – Nix Overview: https://nixos.org/explore/ Nix Tutorial: https://nix.dev/tutorials/first-steps/ Nix Flakes: https://nixos.wiki/wiki/Flakes The Nix Package List: https://search.nixos.org/packages Garnix.IO: https://garnix.io/ Julian’s NixCon Talk, Call by Hash: https://www.youtube.com/watch?v=fU9ogB9hZZA Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
Graphite: Image Editing as a Syntax Tree (with Keavon Chambers & Dennis Kobert) 1:17:32

9 weeks ago1:17:32

1:17:32

Graphite is a new image editor with an interesting architecture - it’s a classic UI-driven app, an image-manipulation language, and a library of programmable graphics primitives that any Rust coder could use, extend or add to. The result is something that you can use like Photoshop or Inkscape, or make use of in batch pipelines, a bit like ImageMagick. Joining me to discuss it are Keavon Chambers & Dennis Kobert, who are hammering away on building a project that’s potentially as demanding as Photoshop, but with a more ambitious architecture. How can they hope to compete? Perhaps in the short term by doing what regular image And is the future of image editing modular? – Graphite Homepage: https://graphite.rs/ Graphite Web Version: https://editor.graphite.rs/ Graphite on Github: https://github.com/GraphiteEditor/Graphite Signed Distance Fields: https://jasmcole.com/2019/10/03/signed-distance-fields/ Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
ReScript: A Better Typed JavaScript? (with Gabriel Nordeborn) 1:32:53

12 weeks ago1:32:53

1:32:53

ReScript is a strongly-typed programming language that compiles to JavaScript, and that puts it squarely in competition with TypeScript. So why would a JavaScript developer choose to learn it next? What does it offer that makes it a tempting proposition? And how are the ReScript developers making life easier for anyone who wants to make the switch? To answer all these questions and more, I’m joined this week by Gabriel Nordeborn, one of ReScript’s compiler contributors. -- ReScript: https://rescript-lang.org/ ReScript & React: https://rescript-lang.org/docs/react/latest/introduction Reanalyze: https://github.com/rescript-lang/reanalyze Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Gabriel on BlueSky: https://bsky.app/profile/did:plc:c55mqp6e6r24rrrypkmx7ker Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
A universal query engine in Rust (with Predrag Gruevski) 1:15:31

14 weeks ago1:15:31

1:15:31

Trustfall is a library based on a simple question - what happens if we can query absolutely anything? If you could join REST APIs and databases with filesystems and dockerfiles? It’s possible in theory because those are all just datasources. Predrag Gruevski is trying to make it easy by building a universal query engine, with pluggable datasources, all in Rust. This week we dive into Trustfall to figure out how it works. How do you model nearly anything as a datasource? How do you make it easy to extend? And what does it take to optimize a query that’s going to be spread out over multiple systems and potentially multiple servers? Questions, questions, questions - all about the act of asking our systems questions. 😃 – Trustfall Source: https://github.com/obi1kenobi/trustfall Cargo Semver Checks: https://crates.io/crates/cargo-semver-checks How to Query Almost Everything: https://predr.ag/querying/ SemVer In Rust (talk): https://predr.ag/blog/semver-in-rust-tooling-breakage-and-edge-cases/#cargo-semver-checks-lints-are-database-queries-in-disguise Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ Kris on Twitter: https://twitter.com/krisajenkins…

Developer Voices

1
Raspberry Pi Hardware & A Lisp Brain (with Dimitris Kyriakoudis) 1:34:43

16 weeks ago1:34:43

1:34:43

Dimitris Kyriakoudis is a researcher, programmer and musician who's combining all three talents to build dedicated music hardware. Specifically a device called the µseq, which reads Lisp programs and uses them to drive synthesizers to make music. In this episode we go through the full platform that he's building, from soldering resistors to an RPi chip, up through writing a Lisp interpreter, to the design ideas that make Lisp a good choice for composing both software and music. – uSeq Homepage: https://www.emutelabinstruments.co.uk/useq/ Emute Lab’s Homepage: https://www.emutelab.org/ Buy a uSeq: https://www.signalsounds.com/emute-lab-instruments-useq-live-coding-voltage-generator-eurorack-module/ Build a uSeq (DIY Kit): https://www.thonk.co.uk/shop/emute-lab-useq/ SICP (book): https://en.wikipedia.org/wiki/Structure_and_Interpretation_of_Computer_Programs Machina Bristronica (expo): https://machinabristronica.uk/ Sonic Pi: https://sonic-pi.net/ Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ Kris on Twitter: https://twitter.com/krisajenkins – 0:00 Intro 2:20 What is µseq? 5:40 Live Coding As Another Instrument 17:42 Why Choose Lisp? 25:03 Different Dialects For Different Musical Tasks? 32:34 Live Coding As Academic Research 44:11 How Do You Fabricate Production Hardware? 49:00 The Triple-E Triangle 1:09:53 How Well Has This Theory Worked Out? 1:20:01 What's This Like To Play Live? 1:25:17 Comparisons With Sonic Pi 1:33:06 Outro…

Developer Voices

1
Software Systems Aren't Just Software (with Diana Montalion) 1:50:14

17 weeks ago1:50:14

1:50:14

If you want to build really large software systems well, you have to stop thinking of them as just software systems. Beyond a certain size, everything your software touches becomes part of the wider system. You’re part of the system, your users are part of the system, and every other employee & department & priority eventually forms part of that system. And that can make it incredibly difficult to make changes, or even to understand which changes will actually matter. That might be overwhelming, but there's hope. Understanding how systems work and how to improve them is something that can be learnt, and improved at. So in the pursuit of better systems understanding, I’m joined this week by Diana Montalion, coder, architect, and author of Learning Systems Thinking. She takes us through how she sees systems, and how we can get better at discovering and understanding our own systems, as we both chew through some difficult systems we’ve worked on in the hope of figuring out how to do it better next time… – Learning Systems Thinking (book): https://www.amazon.co.uk/Learning-Systems-Thinking-Essential-Professionals-ebook/dp/B0D9KWZRT2 Diana’s Website: https://montalion.com/ Scientific Management (Tailorism): https://en.wikipedia.org/wiki/Scientific_management Jay Wright Forrester: https://en.wikipedia.org/wiki/Jay_Wright_Forrester Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Diana on LinkedIn: https://www.linkedin.com/in/dianamontalion/ Diana on YouTube: https://www.youtube.com/@dianamontalion Kris on BlueSky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
Building Fyrox: A Rust Game Engine (with Dmitry Stepanov) 1:43:45

18 weeks ago1:43:45

1:43:45

To kick off 2025 we’re looking at Fyrox a game engine built in Rust, largely by one person - Dmitry Stepanov. For an individual project, it’s covered an incredible amount of ground, covering the rendering and animation features you’d expect from a game engine, with some features that might surprise you - like Rust scripting support with hot-reloading. As we dive into Fyrox, Dmitry explains what it takes to build a game engine, why he chose Rust (and why he’s happy with the choice), and how one person can hope to build a project of that size. – Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Fyrox Homepage: https://fyrox.rs/ The Fyrox Book: https://fyrox-book.github.io/ Rapier Physics Engine: https://rapier.rs/ The Mine (on Steam): https://store.steampowered.com/app/898980/The_Mine/ Dmitry’s Engine: https://github.com/mrDIMAS/DmitrysEngine GJK Collision Detection Algorithm: https://en.wikipedia.org/wiki/Gilbert%E2%80%93Johnson%E2%80%93Keerthi_distance_algorithm WPF: https://en.wikipedia.org/wiki/Windows_Presentation_Foundation PICO-8: https://www.lexaloffle.com/pico-8.php Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ Kris on Twitter: https://twitter.com/krisajenkins…

Developer Voices

1
Testing TVs At Scale With Elixir (with Dave Lucia) 1:16:05

21 weeks ago1:16:05

1:16:05

Integration testing is always a tricky thing, fraught with problems setting up the right environment and attempting to control the system’s state. That’s particularly true when you’re dealing with a mix of software and hardware, and even worse when you don’t have control of what the hardware can do. This week I’m joined by Dave Lucia of TVLab’s, who’s building systems for testing television software at scale, and it’s a problem that needs a huge variety of techniques to crack it. He’s using cameras, real time video processing, Erlang & Elixir and a host of other tools to make it possible to test a fleet of televisions on demand. Sometimes good systems revolve around a single big idea; this time it’s a large combination of solutions, coordinated by the BEAM, that gets the job done. -- TVLabs: https://tvlabs.ai/ Flipper Zero: https://flipperzero.one ATSC 3.0 “NextGen TV”: https://en.wikipedia.org/wiki/ATSC_3.0 Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
Programming As An Expressive Instrument (with Sam Aaron) 1:50:01

23 weeks ago1:50:01

1:50:01

Sam Aaron is the creator of Sonic Pi, one of the most unusual software platforms you’ll encounter. It’s a live-coding playground for making music. A tool that lets you write code that defines sounds and musical phrases, and build up a hole program that plays anything from a short bleep to a whole nightclub set. And Sam’s creator has been using it live for years, weaving drum & bass nights out of thin air, all driven by the Ruby-esque he writes. In this episode we go through Sam’s career path and design journey as we look at what it takes to make a programming language with enough expressivity and productivity to produce music at the speed of Sam’s imagination. -- Sam’s Sonic Pi Course: https://www.patreon.com/posts/new-introductory-115404746 Sonic Pi: https://sonic-pi.net/ SuperCollider: https://supercollider.github.io/ Overtone: https://github.com/overtone/overtone Power Gloves: https://en.wikipedia.org/wiki/Power_Glove Web Audio API: https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API Tau5: https://www.patreon.com/posts/announcing-sonic-112605951 Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
Elm & The Future of Open Source (with Evan Czaplicki) 50:24

24 weeks ago50:24

50:24

Evan Czaplicki—the creator of the Elm programming language —joins me to discuss the state and future of Elm, the friendly, type-safe functional programming language. On many fronts Elm has been a huge success: it’s been popular with new and seasoned programmers alike; it’s helped push several language ideas into the mainstream; it’s been a key part of several successful software businesses and he even found himself employed as a kind of Language Designer in Residence. And yet, the material rewards of a successful open-source project were…lacking. Was he naive? Can an open-source developer stay true to open-source principles and still make a decent living? Is open source being exploited by commercial software businesses? These topics and more tumble out of what has to be the first question in the podcast: What’s happening with Elm? -- Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join ElmLang: https://elm-lang.org/ The Economics Of Programming Languages: https://www.youtube.com/watch?v=XZ3w_jec1v8 Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/…

Developer Voices

1
Programmers, ADHD, And How To Manage Them Both (with Chris Ferdinandi) 1:39:14

25 weeks ago1:39:14

1:39:14

This week we’re going to look at the most essential piece of firmware in a programmer’s toolkit - the brain. I’m joined by Chris Ferdinandi to explore what it’s like to be a programmer with ADHD. It’s an unusual topic for the channel, but the more I spoke to him, the more I wanted to know what coding is like when your brain is wired differently, how we can work more effectively with people with ADHD, and critically, how you manage coders with ADHD. And the answer to that comes full circle, in understanding how coders with ADHD manage themselves… – ADHDFTW Homepage: https://adhdftw.com/developer-voices/ Do I Have ADHD? https://adhdftw.com/do-i-have-adhd/ Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@developervoices/join Chris on Mastodon: https://mastodon.social/@cferdinandi Chris on BlueSky: https://bsky.app/profile/cferdinandi.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ Kris on BlueSky: https://bsky.app/profile/krisajenkins.bsky.social…

Developer Voices

1
MicroServices For Better And Worse (with Ian Cooper and James Lewis) 47:09

26 weeks ago47:09