Introduction Artificial Intelligence is moving beyond simple prompt-and-response interactions. Earlier AI applications were often built around a single prompt sent to a Large Language Model. The model generated an answer, and the workflow ended there. That works well for summaries, brainstorming, and content generation. But real enterprise problems often need planning, tool usage, validation, collaboration, […]
Every data engineer knows the dread of a process that looks perfect on paper but relies entirely on manual intervention. Not long ago, I was managing a critical regulatory compliance data pipeline for a client. The raw ingredients were complex, multi-format regulatory list updates provided in Excel and CSV files. The destination was Snowflake, where […]
Introduction In modern data platforms, ETL pipelines are rarely independent. They are deeply interconnected—one pipeline’s output becomes another pipeline’s input. In one of our production projects, we faced a classic orchestration challenge. Pipelines were scheduled based on time rather than actual completion i.e real-time problem with predictable execution time of pipelines. This article will walks […]
Like many data engineers, I’ve spent a good chunk of my time dealing with a problem that sounds simple on paper but is messy in reality: reliably moving data from source systems into an analytics platform. In one of my recent projects, I worked on setting up data integration using Airbyte, and this post is […]
Introduction Data engineering, once dominated by manual coding, SQL development, and repetitive operational tasks, is entering a new era. With Generative AI (GenAI), data teams are automating ingestion workflows, accelerating data modeling, writing code faster, improving quality checks, and generating documentation instantly. GenAI isn’t just an add-on—it is fundamentally transforming how modern data platforms are […]
In the modern data ecosystem, speed and efficiency are paramount. Whether you’re building real-time analytics pipelines or scaling distributed systems, the bottleneck often lies in data serialization and transport. Enter Apache Arrow Flight—a high-performance RPC framework designed to move large datasets efficiently using the Arrow memory format. What is Apache Arrow Flight? Apache Arrow […]
Let me tell you about the moment I realized I’d been overcomplicating things for years. I was working on a pipeline in Snowflake. You know the type — a multi-stage transformation process where a few base tables feed into intermediate tables, some reconciliation happens, and eventually it all lands in a final reporting layer. I’d […]
Initial Thoughts Having spent over a decade working in software development and data engineering, I thought, where is AI right now? Is it capable of eliminating the developer or is there still some time? So, I challenged myself with building an Android app. That was an unfamiliar area for me. While it intrigued me, there […]
The State of Code Reviews in Today’s Development Landscape: In today’s fast-moving world of software development, AI has made remarkable progress. It can write code, debug errors, and even help design architectures. But let’s be honest, we’re not quite at a point where AI can take over the entire development process. Human developers are still […]
