Data Engineer

Job Description Roles & Responsibilities Key Responsibilities Data Lakehouse Architecture (Supply Chain) ERP Integration: Architect and deploy ingestion pipelines to extract high-volume transactional data from SAP S/4HANA, Ariba, and PLM systems, ensuring near real-time availability for forecasting models. External Feed Integration: Build connectors for external market intelligence feeds (e.g., S&P Global, Orbis, EcoVadis) to enrich internal procurement data with macroeconomic and geopolitical signals. Unified Data Model: Design and implement a standardized procurement data model and taxonomy across multiple entities, harmonizing fragmented datasets into a cohesive layer for analytics. Unstructured Data Pipelines (LeverEDGE) Complex Ingestion: Engineer pipelines to ingest and process unstructured technical data, including PDF tender documents, CAD metadata, and historical CONOPS, transforming them into vector-ready formats for RAG (Retrieval-Augmented Generation) applications. Vector Database Management: Manage and optimize Vector Databases (e.g., Weaviate) to store embeddings of archival proposals and engineering snippets, ensuring high-speed retrieval for AI drafting assistants. Digital Thread Implementation: Establish data lineage and traceability protocols that link requirements to physical components, supporting the Model-Based Systems Engineering (MBSE) "Digital Thread". Governance & Security Defense-Grade Security: Implement Role-Based Access Control (RBAC), audit logging, and data redaction policies to ensure compliance with export controls and strict on-premise security requirements. Quality Control: Deploy automated data quality frameworks to validate BOM (Bill of Materials) completeness and cost data accuracy before it reaches AI models. Infrastructure Optimization: Optimize pipelines for on-premise GPU clusters and air-gapped environments, ensuring efficiency within existing infrastructure limits. Technical Requirements Core Stack: Expert proficiency in Python, SQL, and modern data engineering frameworks (Apache Spark, Kafka, Airflow). Enterprise ERP: Strong experience extracting data from complex ERP environments, specifically SAP S/4HANA and SAP Ariba. Familiarity with SAP BTP is a plus. Database Technologies: Deep understanding of Data Lakehouse architectures (Databricks/Delta Lake), Relational Databases (PostgreSQL), and Vector Databases (Weaviate/Milvus). Data Pipeline Development: Experience building pipelines for RAG solutions, Conversational agents and classical ML models with tools like dbt, dagster, or prefect DevOps/DataOps: Proficiency with containerization (Docker, Kubernetes) and CI/CD pipelines for deploying data workflows in secure environments. Desired Candidate Profile Professional Qualifications Experience: 5+ years of experience in Data Engineering, with at least 2 years focused on building pipelines for Machine Learning or Generative AI applications in an enterprise setting. Domain Knowledge: Experience in Supply Chain, Manufacturing, or Defense sectors is highly desirable. Ability to understand "Bill of Materials" (BOM) structures and procurement lifecycles. Problem Solving: Ability to navigate the "Governance Collision" between agile data work and rigid systems engineering requirements, ensuring data deliverables meet formal Stage Gate reviews. Collaboration: Proven ability to work alongside Data Scientists and Backend Engineers to define data schemas that support predictive modeling and AI agents. Company Industry IT - Software Services Department / Functional Area IT Software Keywords Data Engineer Get real-time job updates only on our App