25. What can you do with Kinesis Data Streams?

AWS Bites

Eoin and Luciano are back with some more AWS messaging services material! This time we talk about Kinesis Data Streams and everything there’s to know about them! In this episode of AWS Bites, we cover the difference between data streaming and queue or message buses. How Kinesis can let you process large batches of messages in near real time. Why you might use Kinesis as a queue or pub/sub bus in some cases. A few different ways to use Kinesis with the focus on avoiding too much complexity. We also provide some insights based on our experience with Kinesis Data Streams including real use cases, monitoring tips and some gotchas to be aware of.

In this episode, we mentioned the following resources:

- Our introductory episode about what services you should use for events: https://www.youtube.com/watch?v=CG7uhkKftoY

- Amazon Kinesis Producer Library (KPL): https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-kpl.html

- Amazon Kinesis Consumer Library (KCL): https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html

- Kinesis Library wrapper for Node.js: https://github.com/awslabs/amazon-kinesis-client-nodejs

- Kinesis Library wrapper for Python: https://github.com/awslabs/amazon-kinesis-client-python

- Kinesis Data Stream with captures from DynamoDB (for CDC): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html

- Kinesis Data Stream with captures from Aurora: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/DBActivityStreams.Overview.html

- Kinesis Data Analytics: https://aws.amazon.com/kinesis/data-analytics/

- Kinesis Firehose: https://aws.amazon.com/kinesis/data-firehose/

- Must-read on Kinesis: Anahit Pogosova’s two-part blog series: https://dev.solita.fi/2020/05/28/kinesis-streams-part-1.html & https://dev.solita.fi/2020/12/21/kinesis-streams-part-2.html

- Cloudonaut Kinesis vs MSK: https://cloudonaut.io/versus/messaging/kinesis-data-streams-vs-msk/

- Deep Dive on Lambda Consumers for Kinesis / Heeki Park - re:invent 2020 https://www.youtube.com/watch?v=tCYwc7-wwsU

You can listen to AWS Bites wherever you get your podcasts. Find all the links on https://awsbites.com

Do you have any AWS questions you would like us to address?

Leave a comment here or connect with us on Twitter:
- https://twitter.com/eoins
- https://twitter.com/loige