AI Operation Engineer

Job Description Roles & Responsibilities AI Operations Engineer Position Overview We are seeking an AI Operations Engineer to operate and maintain a GPU-accelerated AI inference platform. This is a hands-on technical role responsible for the day-to-day operations of LLM serving infrastructure from bare-metal OS provisioning through production model deployment, monitoring, and capacity planning. The platform serves large language models to internal applications. You will be the dedicated engineer ensuring this infrastructure is reliable, performant, and ready to absorb new models and workloads as requirements evolve. Compensation & Benefits We offer a comprehensive expat/travel benefits package to ensure a seamless transition and comfortable stay. Fully Covered Accommodations: Premium housing/accommodation provided. Meals & Dining: Full food and dining arrangements taken care of by the company. Travel & Commute: Comprehensive coverage of international flight tickets and all local business-related transportation. Visa Sponsorship: End-to-end processing and cost management for work visas and legal documentation. Per Diem Allowance: A daily out-of-pocket allowance (Per Diem) will be provided, with the exact structure and commercial terms finalized upon onboarding/joining. Key Responsibility Infrastructure Provisioning & Automation OS to Production: Provision, harden, and patch vanilla Linux, installing GPU drivers, Python, and inference engines. IaC Automation: Maintain version-controlled infrastructure-as-code for repeatable fleet deployments. Air-Gapped Workflows: Manage offline, restricted-network deployments and local model distribution. Container Management: Configure containerized workloads, database backends, and reverse proxies. LLM Inference Engine Operations Engine Deployment: Deploy and maintain LLM inference serving engines across enterprise GPUs. Multi-GPU Sharding: Configure model sharding based on size, topology, and workload requirements. Quantization & Sizing: Manage quantization trade-offs and calculate GPU memory/cache sizing for optimal concurrency. Feature Config: Set up and troubleshoot tool calling, reasoning modes, and multimodal serving. Model Evaluation: Benchmark and deploy new model releases for quality, accuracy, and throughput. API Gateway & Traffic Management Gateway Operations: Configure routing, load balancing, API keys, token counting, and usage tracking. Support Services: Manage gateway-adjacent database backends, caching layers, and reverse proxies. Traffic Diagnosis: Troubleshoot full-path request issues using HTTP, streaming (SSE), and OpenAI API conventions. Monitoring, Reporting, & Cost Analysis Observability Stack: Own metrics collection, dashboards, and GPU-level telemetry (power, temp, VRAM). Operational Metrics: Report on throughput, latencies (p50/p95/p99), queue depth, and tokens per second. Cost Attribution: Deliver token-level cost analysis and usage reporting per user and application. Alert Management: Define and monitor thresholds for hardware health and capacity degradation. Capacity Planning & Performance Engineering Fleet Scaling: Analyze throughput across GPU configs to guide procurement and fleet planning. Benchmarking: Validate new models and quantization methods before production rollout. Optimization: Resolve performance bottlenecks across drivers, runtimes, and network layers. Developer Collaboration & Prompt Engineering Workload Onboarding: Partner with app teams to onboard workloads onto the inference platform. Prompt & Param Tuning: Recommend prompt and parameter adjustments to optimize token usage and model output. Issue Isolation: Troubleshoot model behavior to isolate prompt/model limits from infrastructure bugs. Fine-Tuning (Foundational) Workflow Support: Understand fine-tuning fundamentals (LoRA/QLoRA) to assist ML teams with dataset and training workflows. Model Deployment: Deploy fine-tuned models and manage their distinct serving requirements. Technical Skills Operating Systems: Linux administration (RHEL family preferred), including kernel tuning, service management, storage, and network configuration. GPU and AI Stack: NVIDIA drivers and toolkits, LLM inference engines, model quantization techniques, multi-GPU parallelism concepts. Networking: HTTP/HTTPS, reverse proxy configuration, TLS/SSL, firewall management, network file systems, and load balancing concepts. Monitoring: Metrics collection, visualization, and alerting using industry-standard observability tools. Automation: Configuration management (Ansible preferred), shell scripting, Python for operational tooling. Version Control: Git with disciplined branching, tagging, and commit practices. Python: Virtual environments, dependency management, and debugging Python-based services. Soft Skills Ability to work independently and take ownership of infrastructure decisions with minimal supervision. Clear technical communication able to document procedures, write runbooks, and explain infrastructure constraints to development teams. Methodical troubleshooting approach: isolate variables, reproduce issues, document root causes and fixes. Comfort operating in environments with data sovereignty requirements and restricted internet access Required Qualifications Experience Minimum 4 years of hands-on experience in Linux systems engineering, with at least 2 years involving GPU infrastructure or ML/AI workloads. Demonstrated experience deploying and operating LLM inference engines in production environments. Strong working knowledge of the NVIDIA GPU software stack: drivers, toolkits, runtime libraries, and common failure modes. Experience with infrastructure-as-code and configuration management tools. Solid understanding of Linux containerization technologies, including rootless operation and service management integration Preferred Qualifications Experience with LLM API gateway or proxy solutions for multi-model routing and management. Familiarity with LLM fine-tuning workflows (LoRA, QLoRA, dataset preparation, evaluation). Experience with air-gapped or restricted-network deployments. Knowledge of Arabic NLP or multilingual model evaluation. Experience scaling GPU infrastructure across growing fleet sizes. Familiarity with cloud GPU providers and bare-metal GPU hosting environments. Understanding of Mixture of Experts (MoE) model architectures and their serving implications Note: We required immediate joiners only. Desired Candidate Profile Minimum 4 years of hands-on experience in Linux systems engineering, with at least 2 years involving GPU infrastructure or ML/AI workloads. Demonstrated experience deploying and operating LLM inference engines in production environments. Strong working knowledge of the NVIDIA GPU software stack: drivers, toolkits, runtime libraries, and common failure modes. Experience with infrastructure-as-code and configuration management tools. Solid understanding of Linux containerization technologies, including rootless operation and service management integration Experience with LLM API gateway or proxy solutions for multi-model routing and management. Familiarity with LLM fine-tuning workflows (LoRA, QLoRA, dataset preparation, evaluation). Experience with air-gapped or restricted-network deployments. Knowledge of Arabic NLP or multilingual model evaluation. Experience scaling GPU infrastructure across growing fleet sizes. Familiarity with cloud GPU providers and bare-metal GPU hosting environments. Understanding of Mixture of Experts (MoE) model architectures and their serving implications Ability to work independently and take ownership of infrastructure decisions with minimal supervision. Clear technical communication able to document procedures, write runbooks, and explain infrastructure constraints to development teams. Methodical troubleshooting approach: isolate variables, reproduce issues, document root causes and fixes. Comfort operating in environments with data sovereignty requirements and restricted internet access Company Industry IT - Software Services Department / Functional Area IT Software Keywords AI Operation Engineer Get real-time job updates only on our App