A scalable data pipeline on Azure handling structured and unstructured sources with AI model integration and monitoring.
Designed and deployed a scalable data pipeline on Microsoft Azure for both structured and unstructured data sources. The solution used Azure Data Factory, Databricks, and PySpark for orchestration, model integration, and monitoring, improving data retrieval efficiency by 40% through workflow automation and query optimization.