Senior Observability Engineer

Systems Engineering / Development / Architecture / Integration



Secondary Location:


Telecommute Options:

Flexible for occasional telework – must be local to work location
Join Our Team
Apply now
right arrow
Share on your newsfeed or with a friend
About Peraton

Peraton is a next-generation national security company that drives missions of consequence spanning the globe and extending to the farthest reaches of the galaxy. As the world’s leading mission capability integrator and transformative enterprise IT provider, we deliver trusted, highly differentiated solutions and technologies to protect our nation and allies. Peraton operates at the critical nexus between traditional and nontraditional threats across all domains: land, sea, space, air, and cyberspace. The company serves as a valued partner to essential government agencies and supports every branch of the U.S. armed forces. Each day, our employees do the can’t be done by solving the most daunting challenges facing our customers. Visit to learn how we’re keeping people around the world safe and secure.


Peraton is seeking a Senior Observability Engineer to join our team of highly skilled, dedicated, and motivated cleared professionals supporting the Department of State (DoS) Consular Affairs Enterprise Infrastructure Operations (CAEIO) Program, for the Bureau of Consular Affairs (CA). The program provides IT Operations and Maintenance to modernize the legacy networks, applications, and databases supporting CA services globally.



  • Perform full lifecycle engineering activities (design, develop, test, implement, maintain) for new and existing observability tools/platforms in an enterprise environment with on-premises and cloud-integrated systems/applications/infrastructure.
  • In support of Observability Platform proof-of-concepts, pilots, and production implementations: Gather functional/business requirements, establish success criteria, and develop use cases.
  • Proactively design telemetry strategies to gain real-time insights and identify potential issues before they escalate.
  • Apply knowledge and experience to the following:
    • Telemetry data collection, analysis, and implementation to derive meaningful insights from different sources including metrics, events, logs, and traces.
    • Distributed systems including those with microservices and hybrid infrastructure (cloud/on-premises) to effectively design telemetry pipelines, build monitoring systems, and implement observability practices.
  • Identify patterns, detect anomalies, and build a holistic understanding of system/application/infrastructure behavior.
  • Collaborate across different observability domains to include Infrastructure, Applications (APM), Networking and close these gaps with the cross-functional skills.
  • Provide technical leadership and support to the following: detecting anomalies, troubleshooting incidents, monitoring system health, optimizing resource allocation, enhancing user experience, supporting compliance and security effort.
  • In accordance with scaled agile framework (SAFe), leverage tools (Jazz/Jira) to track work and backlog.
  • Work with Team Lead and/or Observability Project Manager to prioritize efforts and meet deliverable timelines.
  • Assist in development of observability backup recovery methodologies.
  • Design/develop/deploy capabilities that comply with security/integrity controls and limit operational downtime for maintenance activities.
  • Participate in briefing program leadership, the government customer, and other stakeholders.
  • Leverage approved systems for incident/change management (SNOW), work items (Jazz/Jira), documentation (Confluence), and others.
  • Support adhoc data calls from the government customer in support of observability systems.


Core Work Schedule: First Shift, approximately 7:00 am – 3:30 pm, Monday–Friday.


Work Location: This position is remote. However, the candidate must be local to the DMV area. Employees can work in-person with other team members in Sterling, VA on Wednesdays (location is 5 minutes from Dulles Airport).


Required Qualifications

  • Active SECRET clearance with ability to obtain Top Secret.
  • Five plus (5+) years observability platform/tools experience.
  • Subject matter expertise in telemetry.
  • Familiarity with at least one of the .NET languages (e.g. C#, Visual Basic .NET) and HTML5/CSS3.
  • Familiarity with architecture styles/APIs (REST, RPC).
  • Familiarity with Java programming and/or code debugging/testing.
  • Familiarity with software development lifecycle.
  • Experience with underlying databases used by observability platforms.
  • Familiarity with the ASP.NET framework, SQL Server and design/architectural patterns (e.g. Model-View-Controller (MVC)).


Desired Qualifications

  • Expertise in one or more of the following Observability platforms: Cisco AppDynamics, Datadog, Dynatrace, Splunk, others.
  • COTS Application Development.
  • SQL, Oracle DB, Apache/IIS, RHEL/OL.
  • Bachelor’s degree in STEM field.
  • Observability Platform certifications.


Education and Experience: Minimum of 16 years with HS diploma. Minimum of 12 years with BS/BA; Minimum of 10 years with MS/MA; Minimum of 7 years with Ph.D.

Target Salary Range

$146,000 - $234,000. This represents the typical salary range for this position based on experience and other factors.
SCA / Union / Intern Rate or Range


An Equal Opportunity Employer including Disability/Veteran.

Our Values


At Peraton, our benefits are designed to help keep you at your best beyond the work you do with us daily. We’re fully committed to the growth of our employees. From fully comprehensive medical plans to tuition reimbursement, tuition assistance, and fertility treatment, we are there to support you all the way.

  • Paid Time-Off and Holidays
  • Retirement
  • Life & Disability Insurance
  • Career Development
  • Tuition Assistance and Student Loan Financing
  • Paid Parental Leave
  • Additional Benefits
  • Medical, Dental, & Vision Care
Happy man with his two children