The Data Collection team owns the ingestion pipeline -- starting with a lightweight agent to collect, compress, encrypt and ship the data back to the Sumo Logic cloud. This agent runs on a variety of operating systems including the most popular Windows & Linux variants. They comprise of enterprise IT infrastructure, including servers, personal computers and cloud services. These systems can generate petabytes of platform-specific log data, including Event Logs and Event trace Logs, which must be ingested in near-real time and with a minimal resource footprint.
As a Staff Software Engineer working on our Data Collection team, you will architect, implement, and scale the collection systems to ingest this data locally (from the machine generating the events), remotely (from a different machine within our customers’ infrastructure), and from the cloud (Azure, AWS, GCP, etc...).
You are passionate about handling very large volumes of data on a distributed system. You care about producing clean, elegant, maintainable, robust, well-tested code; you do this as a member of a team, helping the group come up with a better solution than you would as individuals. Ideally, you have experience with performance, scalability, and reliability issues of 24x7 commercial services.
What you will be doing:
Design and implement collection system that is extremely high-volume, fault-tolerant, scalable backend systems that process and manage petabytes of customer data.
Analyze and improve the efficiency, scalability, and reliability of our backend systems.
Write robust code; demonstrate its robustness through automated tests.
Work as a member of a team, helping the team respond quickly and effectively to business needs.
What we need from you:
BS in Computer Sciences or a related discipline (MSc, or PhD is a plus).
Minimum of 8+ years of experience deploying production level code, with a proven track record of ownership and delivery.
Object-oriented experience, for example in Java, Scala, Ruby, or C++.
Understand performance characteristics of commonly used data structures (maps, lists, trees, etc).
Desire to learn Scala, an up-and-coming JVM language (scala-lang.org).
Experience with Win32 API, DCOM, WMI and the Windows Event Viewer.
Experience in multi-threaded programming and distributed systems is highly desirable.
Experience in big data and/or 24x7 commercial service is highly desirable.
Agile software development experience (test-driven development, iterative and incremental development) is a plus.