Data Quality Engineers (Databricks)

Job Description Roles & Responsibilities Job Description: The Data Quality Engineer is responsible for designing, implementing, and maintaining enterprise-scale data quality solutions using Databricks , PySpark , and Delta Lake . This role focuses on building automated data profiling, validation, cleansing, monitoring, and remediation capabilities across modern data platforms while ensuring high levels of data integrity, consistency, and reliability. Working closely with data engineers, data architects, governance teams, business stakeholders, and analytics teams, the Data Quality Engineer will develop scalable data quality frameworks, AI-assisted profiling solutions, reusable rule engines, and automated quality monitoring processes that support trusted data products throughout the organization's data lifecycle. Key Responsibilities: Configure, administer, and maintain the Databricks workspace to support enterprise data quality initiatives. Provision and manage Databricks compute clusters, notebooks, Delta Lake structures, and Unity Catalog integration. Develop AI-assisted data profiling notebooks using PySpark to establish baseline data quality measurements across enterprise datasets. Analyze and measure data quality across key dimensions, including: Completeness Uniqueness Validity Consistency Accuracy Timeliness Design, develop, and maintain a scalable Data Quality Rule Factory using parameterized PySpark templates. Create reusable data quality rules that can be deployed across multiple datasets without manual rule development. Integrate data quality validation into Bronze Silver Gold Delta Lake pipelines. Implement automated quality gates to prevent low-quality data from progressing through data processing layers. Design and develop automated data cleansing pipelines using PySpark transformations. Implement data standardization, deduplication, schema harmonization, and data normalization processes. Develop and deploy MLflow-managed machine learning models for anomaly detection and duplicate identification. Ensure AI-generated recommendations are explainable and support human review before implementation. Design and implement exception handling and quarantine mechanisms for records failing quality validation. Capture detailed exception metadata, including: Failure reason Rule reference Affected data element Processing timestamp Build automated reprocessing workflows for corrected or remediated records. Develop Delta Lake aggregation tables to support enterprise reporting and dashboarding of data quality metrics. Produce quality metrics including: Data Quality Scores Rule Pass Rates SLA Compliance Trend Analysis Configure threshold-based alerts using Databricks SQL Alerts and Azure Monitor integration. Develop predictive analytics models to identify datasets at risk of future quality degradation. Support business and technical teams by performing AI-assisted root cause analysis and identifying recurring data quality issues. Generate prioritized remediation recommendations based on detected quality patterns. Collaborate with data engineers, architects, governance teams, and business stakeholders to improve enterprise data quality. Optimize PySpark jobs and Delta Lake workloads for scalability and performance. Develop technical documentation, operational procedures, and best practices for enterprise data quality management. Participate in code reviews, testing, deployment, and continuous improvement of data quality frameworks. Desired Candidate Profile Bachelor's degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related technical field. Proven experience developing enterprise data quality solutions using Databricks and PySpark . Strong hands-on experience with Delta Lake architecture and Medallion (Bronze, Silver, Gold) data pipelines. Experience building automated ETL/ELT workflows using Databricks. Strong knowledge of data quality principles, profiling methodologies, and validation techniques. Experience designing reusable rule-based validation frameworks and scalable data quality engines. Hands-on experience developing PySpark notebooks and optimizing distributed data processing workloads. Experience implementing automated data cleansing, transformation, deduplication, and schema harmonization processes. Experience working with Unity Catalog for data governance and metadata management. Familiarity with MLflow for machine learning model lifecycle management. Knowledge of machine learning techniques for anomaly detection, duplicate identification, and predictive analytics is highly desirable. Experience implementing exception handling, quarantine processes, and automated data remediation workflows. Familiarity with Databricks SQL Alerts, Azure Monitor, and enterprise monitoring solutions. Strong understanding of relational databases, data warehouses, and modern lakehouse architectures. Experience developing dashboards and reporting datasets for monitoring data quality KPIs. Strong SQL skills and experience working with large-scale structured and semi-structured datasets. Understanding of data governance, metadata management, and enterprise data management best practices. Excellent analytical, troubleshooting, and problem-solving skills. Strong communication skills with the ability to work effectively across technical and business teams. Experience working in Agile or Scrum delivery environments is preferred. Databricks, Microsoft Azure, Apache Spark, or related cloud data engineering certifications are highly desirable. Company Industry IT - Software Services Department / Functional Area IT Software Keywords Data Quality Engineers (Databricks) Get real-time job updates only on our App