Python / Spark / AI Developer

Our client is seeking an experienced Python/Spark/AI Developer to help modernise data platforms by replatforming legacy T-SQL workloads onto Spark and Delta Lake pipelines. The successful candidate will work across data engineering, API development and AI-enabled capabilities, with a strong emphasis on type-safe, well-tested and specification-driven Python development.

Skills Required

• 4+ years of professional Python development experience.
• 2+ years of production experience building Spark or comparable distributed data pipelines.
• Strong Python 3.12 knowledge, including type hints, Pydantic, ABC-based provider patterns and modern packaging tools such as uv or Poetry.
• Experience with Click or similar CLI frameworks.
• Strong Apache Spark/PySpark experience, including DataFrame API, Spark SQL, partitioning, performance tuning and driver/executor architecture.
• Understanding of Spark Connect.
• Production experience with Delta Lake or equivalent lakehouse formats such as Iceberg or Hudi.
• Strong SQL skills and the ability to faithfully translate legacy T-SQL and stored-procedure logic into Spark SQL.
• Strong understanding of hashing, surrogate keys and set-based deduplication.
• Hands-on pytest experience, including fixtures, markers and unit, integration and end-to-end testing.
• Experience developing REST APIs using FastAPI or an equivalent framework.
• Practical experience with Docker and Compose-based development environments.
• Strong Git and CI practices, including PR workflows, linting, type checking and automated test gates .

Should you meet the requirements for this position, please email your CV to Sign in to view email. You can also contact us on 031 350 4019 or alternatively you can visit our website www.stand-outstaffing.co.za

Should you not hear from us within 3 days, please consider your application unsuccessful.