Software Engineer - Real Time Data Platform (Senior / Principal)



Software Engineering
New York, NY, USA
Posted on Tuesday, February 6, 2024

Blackbird.AI helps organizations discover emergent threats and stay one step ahead of real-world harm through our AI-powered Narrative and Risk Intelligence Platform. Our commitment is to prioritize safety and security, providing the tools to identify potential risks and ensure a safer environment proactively. No matter the job or where it’s located, we’re all connected by a shared vision: To lead and enhance the landscape of risk intelligence.

As a Senior Data Engineer reporting to the Director of Engineering, Data Platform, you will not only optimize data pipelines for our advanced real-time streaming cloud-hosted analytics platform but also lead the development of new features and capabilities. Your role will be pivotal in shaping the future of our platform, offering the chance to solve some of the most challenging data problems in the industry.

As the Senior Data Engineer, you’ll have the chance to:

  • Lead and manage the development and optimization of ETL processes for various data formats from social media, news, and web sources.
  • Design and implement robust database systems and develop tools for query and analytic processing, focusing on real-time streaming applications.
  • Conduct performance analysis and empirical studies, making strategic decisions on tradeoffs (e.g., cost vs. throughput/latency).
  • Develop, manage, and oversee the database architecture for our real-time streaming cloud-hosted analytics platform.
  • Spearhead build automation, continuous integration, deployment, and performance optimization efforts, upholding our strict security requirements.
  • Drive the project management, leading the development of new features and capabilities.
  • Design test suites and implement inline instrumentation to ensure data correctness.