End-to-end ELT pipeline for healthcare analytics
An end-to-end data pipeline to extract, transform, and analyze data from public Telegram channels focusing on Ethiopian medical businesses. The pipeline provides actionable insights into medical product trends, pricing, and visual content.
Kara Solutions aims to build a robust data platform to generate insights about Ethiopian medical businesses. This platform leverages data scraped from public Telegram channels to answer critical business questions and enhance data analysis capabilities.
Automated Data Scraping: Extracts messages and images from specified Telegram channels
Robust Data Lake: Stores raw, immutable data for auditing and re-processing
Dimensional Data Warehouse: Implements star schema in PostgreSQL for efficient analytical queries
AI-driven Data Enrichment: Utilizes YOLOv8 for object detection on images
Pipeline Orchestration: Manages entire workflow using Dagster for reliability and monitoring
ELT framework: Extract raw data, load into data lake, transform using dbt
Telegram API integration using Telethon for message and media extraction
PostgreSQL data warehouse with dimensional modeling
Computer vision integration with YOLOv8 for image analysis
FastAPI analytical endpoints for structured data access
Enhanced healthcare analytics through efficient data pipeline management.
Explore the complete implementation, documentation, and code on GitHub.