Observability

you own this product
prerequisites
beginner Python packaging • experience reading and creating architecture diagrams
skills learned
Quix Streams • architecture • Kafka topics • writing to database
Robert Koch and Shane Smith-Sahnow
1 week · 4-6 hours per week · BEGINNER
Included with a Manning Online subscription
catalog / Software Development

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

Look inside

Put on your platform architect hat! You’re a member of the development team at Piper Data Concepts (PDC), and your client is looking to modernize its workflow. An existing benchmarked development environment, made up of Kafka, Python, and Postgres, is at your disposal. Now it’s time to start conceptualizing the new and improved workflow. You’ll use Kafka to create an event-driven data pipeline, review and understand business requirements, use Python Poetry to package the project, write Python code using Quix Streams to communicate with and process Kafka data, and store the consumed data in a PostgreSQL database.

This project is a part of the series Event-Driven Data Pipeline with Python and Kafka.
This project is designed for learning purposes and is not a complete, production-ready application or solution.

project authors

Robert Koch
Rob Koch is a Principal Data Engineer at Slalom Build, and one of the community leaders of DeafintheCloud.com. He helps drive cloud-native architecture, blogs about migrating to the cloud and use of Lambdas, and has a passion for data- and event-driven systems. Having earned five AWS certifications (Cloud Practitioner, Big Data Specialty, DevOps Engineer Associate, SysOps Administrator Associate, and Solution Architect Associate), Robert is actively involved in the development community in Denver, often speaking at Denver Dev Day and the AWS Denver Meetup. Robert’s goal is to help the community understand the advantages of migrating to the cloud, being cloud-native, and having “serverless” applications and databases.
Shane Smith-Sahnow
Shane Smith-Sahnow is a software engineer at Netlify. He has spent time at Github and New Relic working on large scale Kafka applications, developing analytical data pipelines, and shipping large scale features.

prerequisites

The liveProject is for learners who want to understand the concepts behind event-driven architecture. Sometimes seeing (and doing it) is believing. To begin these liveProjects you’ll need to be familiar with the following:

TOOLS
  • Visual Studio Code
  • Basic Python
  • Command Window(cmd, powershell, bash, zsh)
  • DBeaver
  • Docker Desktop
  • Apache Kafka
  • Quix Streams
  • Poetry
TECHNIQUES
  • Benchmarking
  • Decoupling
  • Document the environment setup
  • Event-driven architecture
  • Creating Apache Kafka topics
  • Creating and consuming data
  • Writing results to database

features

Self-paced
You choose the schedule and decide how much time to invest as you build your project.
Project roadmap
Each project is divided into several achievable steps.
Get Help
While within the liveProject platform, get help from fellow participants and even more help with paid sessions with our expert mentors.
Compare with others
For each step, compare your deliverable to the solutions by the author and other participants.
books included as resources
Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.

related titles

choose your plan

pro

monthly
annual
$24.99
$249.99
only $20.83 per month
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Observability project for free

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Observability project for free