Python/Spark/AI Developer

Our client is seeking an experienced Python / Spark / AI Developer to build modern data pipelines, APIs and AI capabilities while supporting the replatforming of legacy T-SQL workloads onto Spark and Delta Lake. This role is suited to a developer who is comfortable working autonomously in a specification-driven environment where testing, documentation and production-quality code are essential.

Key Responsibilities

• Build production-grade Spark/PySpark pipelines on Delta Lake.
• Replatform legacy T-SQL logic into Spark SQL while maintaining functional parity.
• Develop modern, type-safe and well-tested Python solutions.
• Build REST APIs using FastAPI, including authentication, idempotency and job-status functionality.
• Validate migrated pipelines against legacy systems and provide parity evidence.
• Work from design documents and Architecture Decision Records (ADRs).
• Document technical changes and maintain appropriate test coverage.
• Develop and maintain CI pipelines with linting, type-checking and automated testing.
• Work within Docker-based development environments.
• Apply sound Git and pull-request practices.

Requirements

• 4+ years’ professional Python development experience.
• At least 2 years’ experience building Spark or comparable distributed data pipelines in production.
• Strong Python 3.12 experience.
• Advanced knowledge of type hints, Pydantic, static analysis tools such as pyright/mypy and modern Python packaging using uv or Poetry.
• Experience with Click or similar CLI frameworks.
• Strong production experience with OSS Apache Spark/PySpark.
• Strong understanding of Spark DataFrame APIs, Spark SQL, partitioning, performance tuning, driver/executor architecture and Spark Connect.
• Experience with Delta Lake or equivalent technologies such as Iceberg or Hudi.
• Strong SQL skills and ability to interpret and faithfully re-express legacy T-SQL logic in Spark SQL.
• Experience with pytest, including fixtures, markers and tiered testing suites.
• Experience developing REST APIs using FastAPI or equivalent.
• Strong Docker and Compose experience.
• Git and CI experience, including linting, type-checking and test gates.
• Bachelor’s degree in Computer Science, Engineering or equivalent experience.
• Ability to work independently from written requirements and design documentation.

Advantageous Experience

• LLM integration and AI feature development.
• Provider-neutral AI architectures and structured output validation.
• Local/self-hosted inference technologies such as Ollama, vLLM, llama.cpp or LM Studio.
• Data governance, privacy engineering and PII protection.
• Hive Metastore, Trino and Apache Ranger.
• Azure ADLS Gen2, Synapse/ADF, Azure Key Vault and Service Bus.
• OpenTelemetry and OpenLineage.
• Migration and parity testing experience.
• Databricks Certified Developer for Apache Spark or an equivalent credential.

Should you meet the requirements for this position, please email your CV to Sign in to view email. You can also contact us on 031 350 4019 or alternatively you can visit our website https://stand-outstaffing.co.za

Correspondence will only be conducted with short listed candidates. Should you not hear from us within 4 days, please consider your application unsuccessful.