- prerequisites
- experience with grouping records in SQL • basics of generating files
- skills learned
- end-to-end pipeline • Kafka data into the database • web scraping • creating reports
pro $24.99 per month
- access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
- choose one free eBook per month to keep
- exclusive 50% discount on all purchases
- renews monthly, pause or cancel renewal anytime
lite $19.99 per month
- access to all Manning books, including MEAPs!
team
5, 10 or 20 seats+ for your team - learn more
Look inside
As a member of the development team at Piper Data Concepts, you’ll carry out the final steps of a workflow-improvement project: enabling your client’s staff to gather workflow process information in real-time. Several prototypes have been built, and the client’s workflow is more resilient than ever. You’ll write Python code that consumes messages from Kafka and prepares them for storing in the database, create Postgres queries to access the aggregated data, and build reports in CSV files to be read by visualization tools—and ultimately, your client’s staff. When you’re done, your client’s modern system will provide a feedback loop, enable external API access to status updates, and be ready for more specialized services to be plugged in later, with no code changes.
This project is a part of the series Event-Driven Data Pipeline with Python and Kafka.
This project is designed for learning purposes and is not a complete, production-ready application or solution.
prerequisites
The liveProject is for those who are interested in adding experience working with event-driven development. This is open to most skillsets and the user needs to have a programming background and be very comfortable with installing server-based software. We will use Python, but it is not a requirement you need to understand every aspect of the language. Thus, why we’re using Poetry to scaffold your Python project and using Quix Streams to handle events from Apache Kafka. To begin these liveProjects you’ll need to be familiar with the following:
TOOLS- Basic Python
- Apache Kafka
- Quix Streams 3.20+
- Poetry
- Git
- PostgreSQL
- Visual Studio Code
- Docker Desktop
- Python library management with Poetry
- End-to-end event-driven pipeline execution
- SQL aggregation and reporting
- Generating CSV reports from streaming data
features
- Self-paced
- You choose the schedule and decide how much time to invest as you build your project.
- Project roadmap
- Each project is divided into several achievable steps.
- Get Help
- While within the liveProject platform, get help from fellow participants and even more help with paid sessions with our expert mentors.
- Compare with others
- For each step, compare your deliverable to the solutions by the author and other participants.
- books included as resources
- Get full access to select books for 90 days. Permanent access to excerpts from Manning products are also included, as well as references to other resources.
related titles
related titles
choose your plan
pro
monthly
annual
$24.99
$249.99
only $20.83 per month
- access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
- choose another free product every time you renew
- choose twelve free products per year
- exclusive 50% discount on all purchases
- renews monthly, pause or cancel renewal anytime
- renews annually, pause or cancel renewal anytime
-
![]()
Automate Reports project for free
Automate Reports project for free
team
monthly
annual
$49.99
$499.99
only $41.67 per month
- five seats for your team
- access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
- choose another free product every time you renew
- choose twelve free products per year
- exclusive 50% discount on all purchases
- renews monthly, pause or cancel renewal anytime
- renews annually, pause or cancel renewal anytime
-
![]()
Automate Reports project for free
