Senior Data Engineer - Python & Microservices

Job Description Roles & Responsibilities We are looking for an experienced Senior Python Data Engineer with strong expertise in Python development, microservices architecture, data engineering, and backend API development. The ideal candidate will have hands-on experience building scalable Python-based applications and data services, designing robust REST APIs, performing complex data transformations, and developing high-performance backend frameworks to support enterprise data-driven applications. This role requires strong proficiency in Python web frameworks, asynchronous programming, microservices architecture, ETL processes, database management, and modern software engineering best practices. Exposure to big data technologies, DevOps practices, and cloud-native architectures will be highly advantageous. Key Responsibilities Design, develop, and maintain scalable Python-based web frameworks and backend services. Develop robust RESTful APIs using Python frameworks such as FastAPI, Flask, or Django. Build and maintain microservices-based applications supporting enterprise data platforms. Design and implement data transformation pipelines for data ingestion, processing, and enrichment. Develop backend systems to efficiently serve datasets through APIs and other interfaces. Collaborate with cross-functional teams to deliver scalable and secure data-driven applications. Design and maintain relational and NoSQL database schemas. Develop efficient ETL (Extract, Transform, Load) processes for enterprise data requirements. Optimize application and database performance to ensure scalability and reliability. Implement asynchronous programming techniques using asyncio and optimize concurrent connections and I/O operations. Ensure data integrity, consistency, and quality across data transformation pipelines. Participate in code reviews and contribute to software engineering best practices. Troubleshoot, debug, and resolve production issues across applications and data pipelines. Collaborate closely with stakeholders and engineering teams throughout the SDLC. Required Technical Skills Core Python Development Strong hands-on expertise in Core Python. Experience building enterprise-grade Python applications. Strong understanding of: Python programming concepts, Object-oriented programming, Asynchronous programming (asyncio), Concurrent connections handling, I/O optimization techniques, Clean code and software design principles Python Web Frameworks Hands-on experience with one or more of the following: FastAPI, Flask, Django, Pyramid Strong understanding of: RESTful API development, Routing, Authentication mechanisms, Testing frameworks, Database integrations, Framework architecture and scalability Microservices Architecture Strong experience with: Microservices-based application design, Service-oriented architectures, API integrations, Distributed systems, Enterprise backend services Knowledge of: API Gateways, Service communication patterns, Scalable backend architectures Security & API Management Experience implementing: OAuth, JWT Authentication, API Security Best Practices, Authorization mechanisms, Encryption standards, Secure API development practices Data Engineering & Data Transformations Strong hands-on experience with: Data transformation pipelines, ETL processes, Data cleansing and enrichment, Data aggregation techniques, Data quality and integrity management Experience in: Dataset preparation, Data processing workflows, Enterprise data platforms Database Management Experience working with: Relational Databases (PostgreSQL, MySQL, SQL Server (preferred)), NoSQL Databases (MongoDB, Redis (good to have)) Strong knowledge of: Database schema design, Performance optimization, Transactions management, Efficient data retrieval techniques ORM Frameworks Hands-on experience with: SQLAlchemy, Django ORM Ability to perform: CRUD operations, Query optimization, Transaction handling, Data modelling DevOps & CI/CD Experience with: CI/CD pipeline development and maintenance, Git-based workflows, Version control best practices, Branching strategies, Code reviews, Production deployments Big Data Technologies (Preferred) Exposure to: Apache Spark, Hadoop, Apache Kafka Understanding of: Data warehousing concepts, Large-scale data processing architectures Scripting Skills Strong proficiency in: SQL, Shell Scripting, Python scripting Cloud & Containerization (Good to Have) Exposure to: AWS, Microsoft Azure, Google Cloud Platform (GCP), Docker, Kubernetes Data Governance (Good to Have) Knowledge of: Data governance principles, Data security best practices, Compliance frameworks, Enterprise data management practices Preferred Additional Skills Experience with R Programming Language. Experience working on enterprise-scale data engineering initiatives. Exposure to cloud-native architectures and containerized deployments. Required Competencies Strong analytical and problem-solving skills. Excellent communication and stakeholder management capabilities. Strong debugging and troubleshooting skills. Ability to work effectively in Agile and collaborative environments. Strong ownership mindset and attention to detail. Ability to adapt quickly to evolving business and technical requirements. Strong documentation and technical communication skills. Desired Candidate Profile Core Python Development: Strong hands-on expertise in Core Python. Experience building enterprise-grade Python applications. Strong understanding of: Python programming concepts, Object-oriented programming, Asynchronous programming (asyncio), Concurrent connections handling, I/O optimization techniques, Clean code and software design principles Python Web Frameworks: Hands-on experience with one or more of the following: FastAPI, Flask, Django, Pyramid. Strong understanding of: RESTful API development, Routing, Authentication mechanisms, Testing frameworks, Database integrations, Framework architecture and scalability Microservices Architecture: Strong experience with: Microservices-based application design, Service-oriented architectures, API integrations, Distributed systems, Enterprise backend services. Knowledge of: API Gateways, Service communication patterns, Scalable backend architectures Security & API Management: Experience implementing: OAuth, JWT Authentication, API Security Best Practices, Authorization mechanisms, Encryption standards, Secure API development practices Data Engineering & Data Transformations: Strong hands-on experience with: Data transformation pipelines, ETL processes, Data cleansing and enrichment, Data aggregation techniques, Data quality and integrity management. Experience in: Dataset preparation, Data processing workflows, Enterprise data platforms Database Management: Experience working with: Relational Databases (PostgreSQL, MySQL, SQL Server (preferred)), NoSQL Databases (MongoDB, Redis (good to have)). Strong knowledge of: Database schema design, Performance optimization, Transactions management, Efficient data retrieval techniques ORM Frameworks: Hands-on experience with: SQLAlchemy, Django ORM. Ability to perform: CRUD operations, Query optimization, Transaction handling, Data modelling DevOps & CI/CD: Experience with: CI/CD pipeline development and maintenance, Git-based workflows, Version control best practices, Branching strategies, Code reviews, Production deployments Big Data Technologies (Preferred): Exposure to: Apache Spark, Hadoop, Apache Kafka. Understanding of: Data warehousing concepts, Large-scale data processing architectures Scripting Skills: Strong proficiency in: SQL, Shell Scripting, Python scripting Cloud & Containerization (Good to Have): Exposure to: AWS, Microsoft Azure, Google Cloud Platform (GCP), Docker, Kubernetes Data Governance (Good to Have): Knowledge of: Data governance principles, Data security best practices, Compliance frameworks, Enterprise data management practices Preferred Additional Skills: Experience with R Programming Language. Experience working on enterprise-scale data engineering initiatives. Exposure to cloud-native architectures and containerized deployments. Required Competencies: Strong analytical and problem-solving skills. Excellent communication and stakeholder management capabilities. Strong debugging and troubleshooting skills. Ability to work effectively in Agile and collaborative environments. Strong ownership mindset and attention to detail. Ability to adapt quickly to evolving business and technical requirements. Strong documentation and technical communication skills. Company Industry IT - Software Services Department / Functional Area IT Software Keywords Senior Data Engineer - Python & Microservices Get real-time job updates only on our App