Senior Data Engineer

Job Description Roles & Responsibilities Key Responsibilities Take ownership of the existing enterprise data lake platform, ensuring scalability, reliability, and performance. Lead the design, architecture, and implementation of cloud-native data lake solutions and integrations. Manage and optimize data ingestion pipelines on Oracle OCI, using tools such as Apache NiFi, Kafka, Batch Processing of data, Data captures, and or CSV. Design and implement pipelines for network data ingestion and file formats (e.g., Parquet, Avro, OCR, etc.), ensuring efficient storage, processing, and retrieval. Build, configure, and tune query engines such as Trino (Presto), Spark, and Hive for efficient analytics and reporting. Implement and maintain metadata management, data governance, and security frameworks. Monitor and troubleshoot system performance, ensuring SLAs are met for ingestion, processing, and query workloads. Automate platform deployment, monitoring, and maintenance with Infrastructure-as-Code (Terraform, CloudFormation, etc.). Collaborate with data engineers, analysts, and business teams to understand data requirements and deliver solutions that maximize data accessibility and usability. Keep the data platform up to date with the latest open-source and cloud-agnostic technologies, implementing upgrades and enhancements where needed. 5+ years of proven, hands-on experience implementing and managing large-scale data lakes in the cloud (OCI). Strong expertise in: Data ingestion & orchestration: Apache NiFi, Apache Kafka, CSV, and others Data processing frameworks: Apache Spark, PySpark, Trino (Presto), Hive, Flink. Storage & lakehouse architectures: Delta Lake, Apache Hudi, Iceberg, and cloud-native object storage (S3). Query & analytics tools: Trino/Presto, SparkSQL, Metabase, or Apache Superset. Experience with data lake file formats such as Apache Parquet, Avro, ORC, CSV, etc. including ingestion, parsing, and analytics within a data lake. Solid understanding of data governance, lineage, cataloging, and security frameworks (Apache Atlas). Experience with CI/CD and IaC (ArgoCD, Terraform, Ansible) for automated deployments. Hands-on experience with cloud security best practices, including IAM, encryption, and network security. Strong proficiency in Python or Java for data engineering and automation tasks. Proven ability to work independently, quickly understand existing environments, and deliver results without extensive training. Preferred Skills Exposure to machine learning workflows integrated with data lakes. Experience with real-time streaming data pipelines. Familiarity with containerization and orchestration (Docker, Kubernetes). Knowledge of cost optimization strategies in cloud-based data platforms. Desired Candidate Profile 5+ years of proven, hands-on experience implementing and managing large-scale data lakes in the cloud (OCI). Strong expertise in: Data ingestion & orchestration: Apache NiFi, Apache Kafka, CSV, and others Data processing frameworks: Apache Spark, PySpark, Trino (Presto), Hive, Flink. Storage & lakehouse architectures: Delta Lake, Apache Hudi, Iceberg, and cloud-native object storage (S3). Query & analytics tools: Trino/Presto, SparkSQL, Metabase, or Apache Superset. Experience with data lake file formats such as Apache Parquet, Avro, ORC, CSV, etc. including ingestion, parsing, and analytics within a data lake. Solid understanding of data governance, lineage, cataloging, and security frameworks (Apache Atlas). Experience with CI/CD and IaC (ArgoCD, Terraform, Ansible) for automated deployments. Hands-on experience with cloud security best practices, including IAM, encryption, and network security. Strong proficiency in Python or Java for data engineering and automation tasks. Proven ability to work independently, quickly understand existing environments, and deliver results without extensive training. Company Industry IT - Software Services Department / Functional Area IT Software Keywords Senior Data Engineer Get real-time job updates only on our App