Find the latest news & updates on AWS

What’s new in AWS Kinesis?

Amazon Kinesis Data firehose now also supports dynamic partitioning, where it continuously groups in transit data using dynamically or statically defined data keys and delivers the data into the individual amazon S3 prefixes by key. This reduces time to insight by minutes, it also reduces the cost, and overall simplifies the architecture.

Working with Streaming Data

For working with streaming data using Apache Flink, we also have AWS kinesis data analytics service, as with Amazon Kinesis Datastream like Kinesis Data Firehose, this service is also a fully managed, serverless, Apache Flink environment to perform stateful processing with sub-second latency. It integrates with several AWS services, supports custom connectors, and has a notebook interface called KDA Studio (Kinesis Data Analytics Studio), a managed Apache Zeppelin notebook, to allow you to interact with streaming data.

Similar to Kinesis Data Analytics for Apache Flink, Amazon managed stream for Apache Kafka or MSK is a fully managed service for running highly available, event-driven, Apache Kafka applications.

Amazon MSK operates, maintains, and scales Apache Kafka clusters, provides enterprises with security features and supports Kafka connect, and also has multiple built-in AWS integrations.

Architecture for Real-Time Reporting

Here we derive insights from input data that are coming from diverse sources or generating near real-time dashboards. With the below architecture what you are seeing is, that you can stream near real-time data from source systems such as social media applications using Amazon MSK, Lambda, and Kinesis Data Firehose into Amazon S3, you can then use AWS glue for Data Processing and Load, Transform data into Amazon redshift using an AWS glue developed endpoint such as an Amazon Sagemaker Notebook. Once data is in Amazon Redshift, you can create a customer-centric business report using Amazon Quick sight.

This architecture helps in identifying an act on deviation from the forecasted data in near real-time. In the below architecture, data is collected from multiple sources using Kinesis Data Stream, it is then persisted in Amazon S3 by Kenisis Data firehose, initial data aggregation, and preparation is done using Amazon Athena and then stored in the AWS S3. Amazon Sagemaker is used to train a forecasting model and create behavioral predictions. As new data arrives it is aggregated and prepared in real-time by Kinesis Data Analytics. The results are compared to the previously generated forecast, Amazon Cloud Watch is used to store the forecast and actual value as metrics, and when actual value deviates and cloud watch alarms trigger an incident in AWS Systems Manager, Incident manager.

Real-time reporting

Architecture for Monitoring Streaming Data with Machine Learning

This architecture helps in identifying an act on deviation from the forecasted data in near real-time. In the below architecture, data is collected from multiple sources using Kinesis Data Stream, it is then persisted in Amazon S3 by Kenisis Data firehose, initial data aggregation, and preparation is done using Amazon Athena and then stored in the AWS S3. Amazon Sagemaker is used to train a forecasting model and create behavioral predictions. As new data arrives it is aggregated and prepared in real-time by Kinesis Data Analytics. The results are compared to the previously generated forecast, Amazon Cloud Watch is used to store the forecast and actual value as metrics, and when actual value deviates and cloud watch alarms trigger an incident in AWS Systems Manager, Incident manager.

Monitoring streaming data

Conclusion

The key considerations, when working with AWS Streaming Services and Streaming Applications. When you need to choose a particular service or build a solution

Usage Patterns‍

Kinesis Data Stream is for collecting and storing data, and Kinesis Data Firehose is primarily for Loading and Transforming Data Streams into AWS Data Stores and Several Saas, endpoints. Kinesis Data Analytics essentially analyzes streaming data.

Throughput

Kinesis streams scale with shards and support up to 1Mb payloads, as mentioned earlier, you have a provisioning mode and an on-demand mode for scaling shard capacity. Kinesis firehose automatically scales to match the throughput of your data. The maximum streaming throughput a single Kinesis Data Analytics for SQL application can process is approximately 100 Mbps.

Latency

‍Kinesis Streams allows data delivery from producers to consumers in less than 70 milliseconds.

Ease of use and cost

‍All the streaming services on AWS are managed and serverless, including Amazon MSK serverless, this allows for ease of use by abstracting away the infrastructure management overhead and of course, considering the pricing model of each service for your unique use case.

‍

About bNoteable

bNoteable helps you showcase your hard work on a path to reach your goals by leveraging your band, orchestra, or vocal experience to its fullest potential to college admissions boards.
This begins early by setting a course that allows you to turn those hours of fun and friendship into leadership experience, hours of practice and performances into scholarship potential, and years of music classes into overall higher SATs and GPA scores, and academic achievement.

Executive Summary

Continuing the development of a musician networking platform which involved implementing new features, enhancing the existing ones, and fixing bugs/errors/issues in the platform by improving its efficiency and productivity along with making the platform responsive.

Problem Statement

Our client wanted us to design and create a social platform where each and every user is able to connect and interact with one another easily. He came to us after a bad experience with some other company and was expecting to continue the development by improving website performance as well as efficiency.
The platform had various bugs which needed to be fixed and some major features were to be added like payment service, OTP service, adding more security along with improving existing features. Performance of platform was being affected as there were some major issues like:
1. Deployment architecture- Everything was deployed on a single EC2 instance due to which there was a high amount of downtime. The performance was impacted more when the user base was increased.
2. The videos on his platform were taking a lot of time to load.

Our Solutions

1) We followed MVC architecture for developing REST API using express as middleware and mongoose for managing data in MongoDB. Authenticated API with jwt by using JSON web token package.
2) Added payment service in the platform by integrating stripe payment gateway with help of stripe package, created OTPs for security/validation which was communicated via SMS with help of Twilio.
3) To improve the performance, we deployed the backend on a separate ec2 instance with Nginx as reverse proxy and pm2 as process manager which comes with a built-in load balancer and helps to keep the application alive forever.
4) Installed Nginx on the server, and changed the Nginx.conf file configurations as per the requirement and it worked as a load balancing solution. Also replaced the lets encrypt SSL certificates with ACM(AWS Certificate Manager) to make certificate renewal, provision, and management process better as well as easy.
5) For adding new features to the platform, the frontend involved creating several components, services, directives, pipes, and modules in Angular.
6) To reduce the load time we implemented Lazy loading with help of Lazy load routes. The reason behind increased load time for videos was the use of video tag over secured protocol, to solve this we used iframe for rendering videos which proved to be much faster.
7) Changed the existing deployment architecture and moved the front-end to S3 so that load on the server can be reduced. We moved the front-end to S3 with CloudFront as CDN for speeding up the distribution of web content and improving performance.

Technologies

Angular 10, Node, Express, MongoDB, AWS S3, EC2, CloudFront

Success Metrics

1. Provided all the deliverables within the expected deadlines, improved performance as down time reduced and videos were no longer buffering for a long time.
2. Met all the expectations of the client and with positive feedback. All his meetings with directors and students were successful due to which he wanted us to implement some more new features on his platform.
3. Continuous reporting of progress to the client.

‍

Executive Summary

Enklu aims to provide an Augmented Reality (AR) runtime for UWP, WebGL, Windows Standalone, Android, and iOS. It carves a niche in the market by providing a product that is highly iterative in that it provides instant feedback to users for changes in layout, assets UI or scripts by automatically downloading new data eliminating the need to rebuild. Enklu is truly cross platform, not only does it compile flawlessly to multiple targets, it also allows for tailoring experiences to multiple platforms. Enklu employs Unity along with a C# and Node.js framework for backend to provide a web app that can help content creators create an AR VR experience. It employs React for its frontend.

Problem Statement

Most of the tech stack was deployed on azure VMs. However, they were using archaic deployment processes with a lot of manual input, coupled with poor infra planning had resulted in a high amount of downtime.
This problem was brought into sharp relief when their user base climbed tenfold. The problem was further compounded by a lack of health checks and resource monitoring. Subpar patches to this had brought the core maintenance and enhancement operations to a screeching halt.

Our Solutions

1) The first thing that we proposed to do was to move the frontend build files to S3 in order to reduce the load on the server, post which we moved on to automating the build and deployment of docker images using git actions and terraform and setting up better resource checks by employing the built-in azure triggers.
2) Next, we proposed rewriting parts of code to better handle errors and setting up node clusters with a load balancer to help reduce the load on the primary unity servers, this also helped with reducing downtime since nodes could be safely brought down without affecting the user experience during low traffic hours.

Technologies

C#, Nodejs, AWS(SQS, S3), Azure(VM and load balancer), Unity, .Net, Docker

Success Metrics

1. Reduced down time
2. Better error alerts
3. Reduced first response time (FRT) for resource hiccups

‍

Project Summary

Creating a Marketing website using ReactJS and AWS for the client to showcase what they do and how they do.
Feature enhancement in an existing web application where people with disabilities can request a communication facilitator or a support service provider and providers can accept a request and receive payment.

Problem Statement

The client divided the project into several MVPs.

As part of MVP-1, the client wanted to create a marketing website that is fast, secure, and allows people to understand what Mizaru is and how it can benefit them. They wanted a website that performs operations faster, is secure from the bots, and is cheaper to maintain.

MVP-2 involved enhancing the client’s existing web application, which was previously very basic. They wanted to implement features like admin dashboard management, QR code-based check-in and check-out of providers to provide service, etc.

In MVP-3 they wanted us to create a mobile application to perform the same functionality.

Our Solutions

1) We created a marketing website for the users using ReactJS. This provides us with a faster way to create and serve the application.
2) For deployment and maintenance, we used AWS. It reduced our cost and maintenance efforts.
3) For enhanced security from bots, we’ve implemented google ReCaptcha v3.
4) Once the user has a clear understanding, they are moved to a web app or a mobile App.
5) Through the web app customers (People with disability) can create a request based on their requirements (e.g. Need a communication facilitator or support service provider). Our application provides a way for people with disabilities to connect with service providers. This request will be visible to multiple service providers in the network and they can choose to accept or reject the request.
6) We integrated a payment gateway for processing the payment. Also, both customers and providers get notified of the multiple events. We created a dashboard for Admins to see the track of various requests and generate reports as per their needs.

Technologies

Express JS, React JS, Redux, AWS, GIT, Hubspot, Google Recaptcha v3

Success Metrics

1. Created and delivered marketing website within the given timeframe.
2. Created report generation feature for admin.
3. Implementation of QR code based check-in and check-out of provider.
4. Email reminders for customer and providers before service.

‍

Project Summary

PXL is an open source social network platform for content creators. It enables users to create public or private spaces for any use such as any particular task base space or any other. Furthermore, users can take advantage of social features such as building connections, posting projects that pique the interest of other users, adding team members, notifications, project participation, and more. They can also manage their profiles and conduct a global search. This social network tool offers an online version where anyone can experience this free tool. PXL’s user interface is very logical, and users can easily navigate through various elements.

Problem Statement

The client’s requirement was to build a full-fledged backend application that can easily integrate with their prebuilt front-end application, and he later asked us to integrate the backend with the front-end.

We had to design and create a social platform where users can showcase their inventions and gain exposure. One can post any software project, categorize them, invite team members, and also participate in other projects.

Additionally, to meet the need for significant content uploads, a solution had to be developed that could easily handle the upload of media files while still being affordable and effective.

We also had to create a real-time notification system that monitors all network activity such as accepting requests, declining requests, and being removed from one’s network.

Our Solution

1. With thorough testing, responsive design, and increased efficiency and performance, we concentrated on completing each task as effectively as we could.
2. Based on the client’s requirements, we used S3 bucket, RDS, EC2, and flask microservice for media files and SES for emails.
   - Amazon S3 was used for file hosting and data persistence.
   – Amazon Relational Database Service (RDS) was used for database deployment as it simplifies the creation, operation, management, and scaling of relational databases.
   – Amazon EC2 was used for code deployment because it offers a simple web service interface for scalable application deployment.
3. We sent emails using Amazon SES because it is a simple and cost-effective way to send and receive emails using your own email addresses and domains.
4. Django-graphQL was used for the backend, and Next.js was used for the front end. Django includes a built-in object-relational mapping layer (ORM) for interacting with application data from various relational databases.
   – GraphQL aims to automate backend APIs by providing type-strict query language and a single API Endpoint where you can query all information that you need and trigger mutations to send data to the backend.
   – Next.js offers the best server-side rendering and static website development solutions. We utilized the flask microservice to help with high content uploads since flask upload files give your application the flexibility and efficiency to manage file uploading and serving.
5. Using Github’s automated CI/CD pipeline we have triggers for code lookup and deployment.

‍

Technologies

Django-GraphQL, Next.js, PostgreSQL, AWS S3, EC2, SES and RDS

Success Metrics

• All deliverables were completed on time and exceeded expectations.
• Met all the expectations of the client and with positive feedback.
• The client was constantly updated on the status of the project.

‍

About bNoteable

bNoteable helps you showcase your hard work on a path to reach your goals by leveraging your band, orchestra, or vocal experience to its fullest potential to college admissions boards.
This begins early by setting a course that allows you to turn those hours of fun and friendship into leadership experience, hours of practice and performances into scholarship potential, and years of music classes into overall higher SATs and GPA scores, and academic achievement.

Executive Summary

Continuing the development of a musician networking platform which involved implementing new features, enhancing the existing ones, and fixing bugs/errors/issues in the platform by improving its efficiency and productivity along with making the platform responsive.

Problem Statement

Our client wanted us to design and create a social platform where each and every user is able to connect and interact with one another easily. He came to us after a bad experience with some other company and was expecting to continue the development by improving website performance as well as efficiency.
The platform had various bugs which needed to be fixed and some major features were to be added like payment service, OTP service, adding more security along with improving existing features. Performance of platform was being affected as there were some major issues like:
1. Deployment architecture- Everything was deployed on a single EC2 instance due to which there was a high amount of downtime. The performance was impacted more when the user base was increased.
2. The videos on his platform were taking a lot of time to load.

Our Solutions

1) We followed MVC architecture for developing REST API using express as middleware and mongoose for managing data in MongoDB. Authenticated API with jwt by using JSON web token package.
2) Added payment service in the platform by integrating stripe payment gateway with help of stripe package, created OTPs for security/validation which was communicated via SMS with help of Twilio.
3) To improve the performance, we deployed the backend on a separate ec2 instance with Nginx as reverse proxy and pm2 as process manager which comes with a built-in load balancer and helps to keep the application alive forever.
4) Installed Nginx on the server, and changed the Nginx.conf file configurations as per the requirement and it worked as a load balancing solution. Also replaced the lets encrypt SSL certificates with ACM(AWS Certificate Manager) to make certificate renewal, provision, and management process better as well as easy.
5) For adding new features to the platform, the frontend involved creating several components, services, directives, pipes, and modules in Angular.
6) To reduce the load time we implemented Lazy loading with help of Lazy load routes. The reason behind increased load time for videos was the use of video tag over secured protocol, to solve this we used iframe for rendering videos which proved to be much faster.
7) Changed the existing deployment architecture and moved the front-end to S3 so that load on the server can be reduced. We moved the front-end to S3 with CloudFront as CDN for speeding up the distribution of web content and improving performance.

Technologies

Angular 10, Node, Express, MongoDB, AWS S3, EC2, CloudFront

Success Metrics

1. Provided all the deliverables within the expected deadlines, improved performance as down time reduced and videos were no longer buffering for a long time.
2. Met all the expectations of the client and with positive feedback. All his meetings with directors and students were successful due to which he wanted us to implement some more new features on his platform.
3. Continuous reporting of progress to the client.

‚Äç