In this episode of "How AI is Built", data architect Anjan Banerjee provides an in-depth look at the world of data architecture and building complex AI and data systems. Anjan breaks down the basics using simple analogies, explaining how data architecture involves sorting, cleaning, and painting a picture with data, much like organizing Lego bricks to build a structure. Summary by Section Introduction Anjan Banerjee, a data architect, discusses building complex AI and data systems Explains the basics of data architecture using Lego and chat app examples Sources and Tools Identifying data sources is the first step in designing a data architecture Pick the right tools to extract data based on use cases (block storage for images, time series DB, etc.) Use one tool for most activities if possible, but specialized tools offer benefits Multi-modal storage engines are gaining popularity (Snowflake, Databricks, BigQuery) Airflow and Orchestration Airflow is versatile but has a learning curve; good for orgs with Python/data engineering skills For less technical orgs, GUI-based tools like Talend, Alteryx may be better AWS Step Functions and managed Airflow are improving native orchestration capabilities For multi-cloud, prefer platform-agnostic tools like Astronomer, Prefect, Airbyte AI and Data Processing ML is key for data-intensive use cases to avoid storing/processing petabytes in cloud TinyML and edge computing enable ML inference on device (drones, manufacturing) Cloud batch processing still dominates for user targeting, recommendations Data Lakes and Storage Storage choice depends on data types, use cases, cloud ecosystem Delta Lake excels at data versioning and consistency; Iceberg at partitioning and metadata Pulling data into separate system often needed for advanced analytics beyond source system Data Quality and Standardization "Poka-yoke" error-proofing of input screens is vital for downstream data quality Impose data quality rules and unified schemas (e.g. UTC timestamps) during ingestion Complexity arises with multi-region compliance (GDPR, CCPA) requiring encryption, sanitization Hot Takes and Wishes Snowflake is overhyped; great UX but costly at scale. Databricks is preferred. Automated data set joining and entity resolution across systems would be a game-changer Anjan Banerjee: LinkedIn Nicolay Gerold: ⁠LinkedIn⁠ ⁠X (Twitter) 00:00 Understanding Data Architecture 12:36 Choosing the Right Tools 20:36 The Benefits of Serverless Functions 21:34 Integrating AI in Data Acquisition 24:31 The Trend Towards Single Node Engines 26:51 Choosing the Right Database Management System and Storage 29:45 Adding Additional Storage Components 32:35 Reducing Human Errors for Better Data Quality 39:07 Overhyped and Underutilized Tools Data architecture, AI, data systems, data sources, data extraction, data storage, multi-modal storage engines, data orchestration, Airflow, edge computing, batch processing, data lakes, Delta Lake, Iceberg, data quality, standardization, poka-yoke, compliance, entity resolution --- Send in a voice message: https://podcasters.spotify.com/pod/show/nicolaygerold/message