Senior AI Test and Evaluation Engineer

Looking for the opportunity to work on problems that matter, with colleagues that share your interest and expertise in AI and Machine Learning?   

Leidos is looking for a Senior  AI Test and Evaluation Engineer to apply their expertise to help Leidos research, develop, and commercialize test and evaluation techniques for AI systems. This includes leading the test and evaluation planning, process definition, tools and testing pipeline research, as well as verification and validation planning.

 We are looking for someone who is intellectually adaptive, likes to collaborate and is inquisitive. You will work alongside junior and senior research/data scientists and data engineers with expertise in machine learning and artificial intelligence, who lead advanced research in AI. 

You will work with a team of data scientists and data engineers to operationalize your research and make resulting tools and technologies deployable into operational environments.  You will contribute to new approaches for test and evaluation of AI models that produce measurable advances over state of the art solutions.  

Primary Responsibilities

• Develop AI Test and Evaluation CONOPS, whitepapers and other documentation
• Develop scalable Test and Evaluation plans for a variety of AI models, to include process, techniques and tools for evaluation
• Design tests for each type of AI model and around risks related to each model
• Works with data scientists, engineers and developers to:
  • understand the AI system under test to develop test procedures
  • identify test data and strategy for data set quality analysis
  • integrate testing tools and procedures with the CI/CD pipeline
• Develop and collect metrics to analyze evaluation process, as well as metrics related to the AI-Model under test
• Collaborate with AI Governance team to maintain visibility and explainability through testing
• Implement testing processes in the AI system design, development and deployment life cycles
• Identify the risks in testing of projects, particularly for assessing the limitations of planned tests on complex AI systems
• Work within teams of AI/ML researchers and engineers using Agile development processes 

Basic Qualifications

• Bachelor's degree in Computer Science, Data Science or related field and 8+ years of relevant experience, Masters with 6+ years experience, or PhD with 4+ years experience.
• Experience with MLOps tools such as Kubernetes, KubeFlow, MLFlow or similar tools.
• Experience using deep learning tools such as Pytorch  and TensorFlow
• Practical experience with:
  • AI Performance and vulnerability assessment
  • Adversarial AI toolkits (e.g., ART, CleverHans)
  • Formal methods for AI model assurance evaluation
  • Applying and automating AI interpretability & explainability tools and methods
  • System and subsystem level test process and automation
  • Creating user acceptance test scenarios
  • SecDevOps tooling and MLOps pipeline development

Preferred Qualifications

• Experience with building LLM and other Generative AI applications
• Experience creating and applying Adversarial AI techniques (e.g., evasion attacks, inversion attacks, poisoning)
• Experience evaluating a broad range of AI model performance and bias/fairness metrics
• Ability and willingness to obtain a Top Secret security clearance 

Original Posting Date:

While subject to change based on business needs, Leidos reasonably anticipates that this job requisition will remain open for at least 3 days with an anticipated close date of no earlier than 3 days after the original posting date as listed above.

Pay Range:

The Leidos pay range for this job level is a general guideline only and not a guarantee of compensation or salary. Additional factors considered in extending an offer include (but are not limited to) responsibilities of the job, education, experience, knowledge, skills, and abilities, as well as internal equity, alignment with market data, applicable bargaining agreement (if any), or other law.