Streamlining data processing and efficiently analyzing data through a data warehouse solution

Project Summary

PXL is an open source social network platform for content creators. It enables users to create public or private spaces for any use such as any particular task base space or any other. Furthermore, users can take advantage of social features such as building connections, posting projects that pique the interest of other users, adding team members, notifications, project participation, and more. They can also manage their profiles and conduct a global search. This social network tool offers an online version where anyone can experience this free tool. PXL’s user interface is very logical, and users can easily navigate through various elements.

Problem Statement

The client’s requirement was to build a full-fledged backend application that can easily integrate with their prebuilt front-end application, and he later asked us to integrate the backend with the front-end.

We had to design and create a social platform where users can showcase their inventions and gain exposure. One can post any software project, categorize them, invite team members, and also participate in other projects.

Additionally, to meet the need for significant content uploads, a solution had to be developed that could easily handle the upload of media files while still being affordable and effective.

We also had to create a real-time notification system that monitors all network activity such as accepting requests, declining requests, and being removed from one’s network.

Our Solution

1. With thorough testing, responsive design, and increased efficiency and performance, we concentrated on completing each task as effectively as we could.
2. Based on the client’s requirements, we used S3 bucket, RDS, EC2, and flask microservice for media files and SES for emails.
   - Amazon S3 was used for file hosting and data persistence.
   – Amazon Relational Database Service (RDS) was used for database deployment as it simplifies the creation, operation, management, and scaling of relational databases.
   – Amazon EC2 was used for code deployment because it offers a simple web service interface for scalable application deployment.
3. We sent emails using Amazon SES because it is a simple and cost-effective way to send and receive emails using your own email addresses and domains.
4. Django-graphQL was used for the backend, and Next.js was used for the front end. Django includes a built-in object-relational mapping layer (ORM) for interacting with application data from various relational databases.
   – GraphQL aims to automate backend APIs by providing type-strict query language and a single API Endpoint where you can query all information that you need and trigger mutations to send data to the backend.
   – Next.js offers the best server-side rendering and static website development solutions. We utilized the flask microservice to help with high content uploads since flask upload files give your application the flexibility and efficiency to manage file uploading and serving.
5. Using Github’s automated CI/CD pipeline we have triggers for code lookup and deployment.

‍

Technologies

Django-GraphQL, Next.js, PostgreSQL, AWS S3, EC2, SES and RDS

Success Metrics

• All deliverables were completed on time and exceeded expectations.
• Met all the expectations of the client and with positive feedback.
• The client was constantly updated on the status of the project.

‍

Project Summary

Creating a Marketing website using ReactJS and AWS for the client to showcase what they do and how they do.
Feature enhancement in an existing web application where people with disabilities can request a communication facilitator or a support service provider and providers can accept a request and receive payment.

Problem Statement

The client divided the project into several MVPs.

As part of MVP-1, the client wanted to create a marketing website that is fast, secure, and allows people to understand what Mizaru is and how it can benefit them. They wanted a website that performs operations faster, is secure from the bots, and is cheaper to maintain.

MVP-2 involved enhancing the client’s existing web application, which was previously very basic. They wanted to implement features like admin dashboard management, QR code-based check-in and check-out of providers to provide service, etc.

In MVP-3 they wanted us to create a mobile application to perform the same functionality.

Our Solutions

1) We created a marketing website for the users using ReactJS. This provides us with a faster way to create and serve the application.
2) For deployment and maintenance, we used AWS. It reduced our cost and maintenance efforts.
3) For enhanced security from bots, we’ve implemented google ReCaptcha v3.
4) Once the user has a clear understanding, they are moved to a web app or a mobile App.
5) Through the web app customers (People with disability) can create a request based on their requirements (e.g. Need a communication facilitator or support service provider). Our application provides a way for people with disabilities to connect with service providers. This request will be visible to multiple service providers in the network and they can choose to accept or reject the request.
6) We integrated a payment gateway for processing the payment. Also, both customers and providers get notified of the multiple events. We created a dashboard for Admins to see the track of various requests and generate reports as per their needs.

Technologies

Express JS, React JS, Redux, AWS, GIT, Hubspot, Google Recaptcha v3

Success Metrics

1. Created and delivered marketing website within the given timeframe.
2. Created report generation feature for admin.
3. Implementation of QR code based check-in and check-out of provider.
4. Email reminders for customer and providers before service.

‍

About

This financial services organization provides financial planning, advice, and educational resources that investors need in a timely manner - including retirement planning, wealth preservation, brokerage services, and more. The organization services customers worldwide to help them realize their financial goals.

‍

Business Challenge

The data ingestion process of a well-known financial services organization was not designed to scale to handle peak and projected future volume levels. Failures in that process could directly impact their ability to provide accurate and timely analysis, reporting, and investment advice.

Were that to happen, the ensuing reduction in assets under management and future opportunity loss would easily total tens of millions of dollars. Additionally, the data came from multiple sources in multiple formats and the whole system was governed by FINRA compliance regulations, including data encryption. 

‍

Technical Challenge

The organization needed an event-driven data processing solution that securely ingests, prepares, analyzes, and scales during peak and projected volume levels. This solution needed to handle data from multiple data sources in multiple formats, be cost-effective, and meet their FINRA compliance regulations. This solution would also allow the organization to have access to timely analysis and reporting with a comprehensive visualization of the data.

‍

AWS Services Adopted for this Solution

• Amazon EventBridge, for data requests
• AWS Lambda, for file validation
• AWS Glue, for ETL
• Amazon Athena, for queries
• Amazon QuickSight, for data visualization
• Amazon CloudWatch and Amazon SNS for monitoring and alerts

‍

‍

Event-Driven Architecture Solution for Financial Services Firm

Cloudtech implemented an event-driven solution anchored by Amazon EventBridge, AWS Lambda, and AWS Glue. During non-working hours, Amazon EventBridge triggers Lambda to request new data and store it in Amazon S3. AWS Lambda also performs file validation before moving on to a AWS Glue Crawler and then AWS Glue Job for ETL to an AWS Glue Catalog, where the data is queried using Amazon Athena and visualized using Amazon QuickSight. Monitoring and alerting are handled throughout the process flow using Amazon CloudWatch and Amazon SNS. The result not only enhances scalability and accuracy, but also enables cost savings.

‍

“Cloudtech's event-driven architecture solution is the backbone of our timely and accurate investment advice that our clients rely on.” 
- VP of Engineering, Financial Services

‍

Event-Driven Architecture Results

Cloudtech provided the organization with an event-driven data processing solution which was secure, scalable, and handled multiple data sources and data formats. In addition, the solution provided timely analysis, reporting and a comprehensive view which allowed the organization to provide accurate investment advice to their clients. In turn, this solution helped the organization avoid the potential future opportunity losses which could have cost them tens of millions of dollars.

‍

Increase Your Data Security

Learn how you can realize the benefits of an event-driven architecture for your financial services data, to improve security, analysis, and customer experience. Contact Cloudtech today.

‍