EN DE
Cloud

Cloud migration of an on-prem Hadoop stack

We helped a leading European security software provider with a global footprint to migrate their on-premises Hadoop stack to a managed SaaS solution on Google Cloud Platform (GCP). In just 12 weeks, we delivered a secure, scalable, and cost-effective solution, leveraging GCP’s Dataproc serverless, cloud storage, and IAM capabilities.

About the client and the problem

  • The client is a leading security software provider for Europe and the Americas, selling security solutions to top banks, retailers, software companies, manufacturing giants, healthcare companies and other major enterprises. Following huge success in the protection of unstructured data, they are looking to make waves in the data classification space.

  • The crucial step in compliance is to understand what constitutes sensitive data, where and how it is stored, and who can access it. Enterprises require a user friendly, performant, reliable and scalable platform to solve this problem of data discovery and classification. Our client’s Data Discovery and Classification(DDC) offering enables organizations to get complete visibility of sensitive data across cloud, big data, and traditional environments.

The primary challenge faced by the client

The client’s existing solution consisted of a management appliance, discovery and scanning agents, along with a Hadoop cluster for data analytics. The Hadoop cluster turned out to be an administrative and cost overhead for customers and needed to be replaced with a managed SaaS component keeping the following in mind:

Transparent solution - agnostic to changes in the platform consuming this service.

Reliable, scalable, performant and fault tolerant solution required.

Encrypted storage on cloud, with tight access control and permissions only for authorized+authenticated sources.

Secure managed service that replaced the entire Hadoop stack they were using or parts thereof.

Reduced total cost of ownership for end customers.

Solution & implementation

One of our client’s strengths is a strong partnership with Google for their cloud solutions. GCP was the cloud of choice. We were already familiar with their DDC codebase. The project proceeded along different dimensions as follows:

Design and architecture

Picked the leanest GCP components that fit best in this project. Per the requirements, GCP’s Dataproc serverless, cloud storage, API gateway, IAM and load balancer components were integral to the design. Several relatively heavy options like GKE(managed kubernetes), GCE(virtual machines for compute) etc. were eliminated. Basic non-functional requirements like scalability, multi-tenancy, multiregion support etc. were also addressed in the architecture.

Security review

A detailed security review was conducted and the threat model for the suggested architecture was analyzed. Quick iterative changes were made to the design based on inputs from the security team. The primary focus was on access control and authorisation.

Quick prototyping and demo

Demo in ~12 weeks. After the first demo, certain non-functional requirements like load balancing, security enhancements, cost analysis etc. were addressed.

Quality assurance

The entire project was managed in features. Each feature was considered complete only after a round of unit testing, manual validation and automated tests checked in to the code repository. This helped build a strong readiness criteria for deployment of the service.

Conclusion

  • This project was managed by Coriolis from end-to-end. Given requirements, several services were offered to the client in this project:

Quality assurance

Architecture

Design

Partner integration

Security verification

Fin-ops for cost monitoring and optimisation

Planning and project management

We are now committed to taking this project forward and helping our clients SaaS-ify core components of their DDC stack so that they can continue to deliver high value, easy to use security products to their customers.

About us

Coriolis Technologies is a leading provider of software engineering solutions, leveraging our deep expertise to drive innovation and deliver value to clients across diverse industries. With a commitment to excellence and customer satisfaction, we continue to pioneer transformative solutions in the realm of cloud computing and data security.

Other case studies