
Senior Principal Machine Learning Engineer (Security/Advanced Threat Protection)
PaloAlto Networks
Santa Clara, CA • 1h ago • On-site
Senior
On-site
190,400 - 308,000 USD/yr
Match your resume with similar open roles
Upload your resume to automatically match with active openings similar to this role.
Potential salary for Senior Principal Machine Learning Engineer (Security/Advanced Threat Protection) role
Improve your resume now
$190,400 – $308,000/year
Quick Facts
- Role: Senior Principal Machine Learning Engineer (Advanced Threat Protection)
- Focus: Chief technical authority for AI/ML innovation and architectural strategy in cloud security (SECaaS)
Description
Lead architectural strategy and design excellence for security cloud systems, defining a multi-year engineering roadmap and guiding foundation model adoption. Architect large-scale telemetry ingestion/feature pipelines and production-grade agentic ecosystems, and take hands-on ownership of frontier model deployment, diagnosis, and remediation in real environments. Partner across product, security research, data science, and infrastructure to translate threat landscapes into executable designs.
Responsibilities
- Set technical direction by evaluating AI research and steering investments toward production-viable architecture
- Drive engineering culture via RFC standards, architectural review boards, and mentoring Staff/Principal engineers
- Lead end-to-end system design for distributed, fault-tolerant cloud systems processing real-time security data streams
- Architect agentic frameworks with dynamic planning, stateful memory persistence, secure tool execution, and deterministic guardrails
- Build data/telemetry pipelines for ingestion, preprocessing, and feature engineering using Apache Spark, SQL, and BigQuery on Google Cloud
- Guide model architecture choices across LLMs, SLMs, and multimodal systems to optimize cost, latency, privacy, and efficacy
- Diagnose production model failures (drift, context issues, alignment collapse, hallucinations) and remediate via fine-tuning and parameter-efficient methods
- Debug distributed GPU training/inference issues including CUDA OOM, NCCL overhead, and kernel performance stalls
Requirements
- 10+ years progressive software engineering and systems design with technical leadership in high-availability enterprise-scale platforms
- Ability to author complex system designs from scratch and manage architectural trade-offs, including legacy-to-modern migrations
- Proven cross-functional alignment across product, security research, data science, SecOps, and infrastructure
- Deep hands-on knowledge of modern AI architectures (Transformers, attention, tokenization) and production experience deploying/serving LLMs and SLMs
- Expert in model triage and remediation pipelines (data curation, synthetic data generation, SFT, DPO, RL, LoRA/PEFT)
- Practical debugging and profiling of distributed GPU workloads (memory leaks, concurrency issues, cluster communication bottlenecks)
- Advanced Python for AI/ML pipelines and scientific computing
- Production-level fluency in Go or Java for low-latency concurrent microservices
- Strong data engineering skills: Apache Spark, distributed processing, SQL, BigQuery, and Google Cloud orchestration (Dataproc, GKE)
- BS/MS/PhD in a relevant technical discipline
Benefits
- Starting base salary range: $190,400.00 - $308,000.00 per year (plus potential restricted stock units and bonus)
- Immigration sponsorship eligibility: Yes
See similar jobs
Explore similar jobs and use your resume score to choose where to apply next
See more
Principal Machine Learning Engineer (Agentic AI for Cloud Security)
PaloAlto Networks
Senior
Technology
CA, United States of America · On-site
$163k - $264k/yr
6 days ago
Principal Software Engineer
Microsoft
Senior
Technology
Redmond, WA · Hybrid
$143k - $275k/yr
13 days ago

Principal Software Engineer & Architect, Cyber Defense Engineering
ServiceNow
Senior
Technology
Santa Clara, CA · Hybrid
N/A
16h ago

Principal Software Engineer
Microsoft Corporation
Senior
Technology
Redmond, WA, United States of America · On-site
$143k - $275k/yr
9 days ago

Principal Machine Learning Engineer (AI)
Next Step Systems
Senior
Technology
Remote
$170k - $200k/yr
2 days ago
Tailor Resume to fit the Job Offer
Use AI to find out how well the skills on your resume fit this job description.

Principal Software Engineer, Cloud Security Platform
PaloAlto Networks
Senior
Technology
CA, United States of America · On-site
$156k - $253k/yr
7 days ago
Principal Software Engineer, AI Frameworks
Microsoft
Senior
Technology
Remote
$143k - $275k/yr
3 days ago

Principal Software Engineer (AI)
ServiceNow
Senior
Technology
Santa Clara, CA · On-site
$221k - $387k/yr
9 days ago

Principal Software Engineer & Architect, Cyber Defense Engineering
ServiceNow
Senior
Technology
Santa Clara, CA · On-site
$221k - $387k/yr
6 days ago
Unlock Full Resume Report
ATS Pass
Missing keywords
Tailored AI suggestions
Job match analysis
Interview-focused insights