Senior System Engineer - Apache Spark DevOps
Shanghai, China

is seeking Senior CI / CD Software Engineers to accelerate Apache Spark and related data science frameworks.

Data scientists spend a considerable amount of time exploring data, iterating over machine learning (ML) experiments. Every hour of compute required to sort through datasets, extract features, fit ML algorithms, hinders the ability of data scientists to drive towards results.

Apache Spark is the most popular data processing engine in data centers for data science. It is used for interactive data science, from data preparation, to running ML experiments, and all the way to deployment of ML applications.

You will work with open source community to accelerate Apache Spark with GPU for data science. Nvidia believes that data science workflows can benefit tremendously from being accelerated, to enable data scientists to explore many more and larger datasets to drive towards their business goals, faster, and more reliably.

What you'll be doing :

Serve technical lead in defining, designing, developing and maintaining the DevOps tools, frameworks & platforms.

Implement, advocate and carry out CICD conventions and write tools to automate various steps involved in this process.

Develop and maintain Build, Deployment, and Continuous Integration infrastructure.

Enable development team by providing automated build and test solutions using Docker, Kubernetes / YARN and on-prem / CSPs.

Engaging open source communities, including Apache Spark and , to develop the CI / CD pipelines.

Work closely with Development and QA teams to help ensure end-to-end quality.

Full stack development opportunities depending on the candidate's capabilities.

What we need to see :

BS or MS in Computer Science, Computer Engineering, or closely related field

5+ years of working experience in software development

2+ years experience in CI / CD system

Good understanding of Quality Control and Test Automation in Agile-based Continuous Integration environments

Strong programming and debugging skills in Python / Java / Go / C# with extensive bash scripting experience.

Excellent knowledge with Gitlab / Github or other source version control systems.

Excellent knowledge on Docker, Jenkins

Excellent knowledge on build tools like MAVEN, SBT and CMake, unit testing and code-coverage tools

Excellent knowledge of various Linux systems like Ubuntu, CentOS.

Experience with configuration management

Excellent programming skills in Python, Shell script

Able to work successfully with multi-functional teams across organizational boundaries and geographies

Highly motivated with strong communication skills

Ways to stand out from the crowd :

Experience with Spark and Spark cluster(like Yarn, Kubernetes) management would be a plus

Excellent knowledge of open source project management would be a plus

We are an AA / EEO / Disabled employer and with highly competitive salaries and a comprehensive benefits package, NVIDIA is widely considered to be one of the technology world’s most desirable employers.

We have some of the most brilliant and talented people on the planet working for us. Are you creative and autonomous? Do you love a challenge?

If so, we want to hear from you.


Thank you for reporting this job!

Your feedback will help us improve the quality of our services.

通过点击 "继续", 我 同意neuvoo处理我的信息数据, 并给我发送电子邮件提醒 详见neuvoo 隐私政策 。我可以在任何时候取消订阅。