Automating Data Collection and Analysis for In-depth Societal Trend Reports: A Solution for a Malaysian News Agency

DataSteroid

Automating Data Collection and Analysis for In-depth Societal Trend Reports: A Solution for a Malaysian News Agency

Overview

TechVariable partnered with a large Malaysian news agency that required a streamlined approach to collect, integrate, and analyze enormous amounts of data from disparate sources. The goal was to generate intelligent insights and reports on key societal trends, which the government and other entities often used. By developing a one-stop solution, TechVariable increased efficiency in data collection, improved accuracy, and enabled real-time reporting.

Challenges

Manual Labor and Inefficiency: Data from various sources, such as social media, newspapers, and physical documents, was manually entered into spreadsheets, which is time-consuming, labor-intensive, and prone to errors.
Data Standardization: The diverse formats, sizes, and languages in which data arrived needed harmonization for efficient analysis.
Limited Processing: Their existing AWS Lambda services had constraints on the number of requests processed simultaneously.
Customization Constraints: Their system lacked access control through role management, and the existing reports did not fully visualize data trends.

Solutions

In response to these challenges, TechVariable created a platform that leveraged the power of DataSteroid with a few special customizations based on the client’s needs to automate the data collection process, harmonize it, and report accordingly.

Pipeline 1: With the help of DataSteroid, we made connectors for online and hardcopy data sources, including PDFs and jpegs, using Optical Character Recognition (OCR). We also built a module to summarize lengthy articles based on identified keywords.
Pipeline 2: We developed custom connectors in DataSteroid to bring in audio-video files. Google APIs were used to process these files from platforms like YouTube.
Pipeline 3: A custom mechanism was developed to segregate hashtags, keywords, stories, etc., from online sources for further processing.

Result

The enhancements have significantly boosted system efficiency and accuracy, automating data collection for increased precision and scalability. With improved processing capabilities, the platform can now handle sentiment analysis of more than 100,000-150,000 posts monthly based on various parameters, reducing labor requirements and enhancing resource efficiency. The system’s new customizable features, including user access control, advanced filters, search capabilities, auto-visualization, and real-time reporting, provide users with profound insights into societal trends, elevating data analysis and reporting efficiency.

Advanced Data ETL, Reporting and Gen AI

No-code data engineering

Automated data transformation

Enterprise-grade LLM

MODULES

Healthcare Data Management

Advanced Data ETL, Reporting and Gen AI

No-code data engineering

Automated data transformation

Enterprise-grade LLM

Solutions that ensure

Advanced Data ETL, Reporting and Gen AI

No-code data engineering

Automated data transformation

Enterprise-grade LLM

Explore more

Advanced Data ETL, Reporting and Gen AI

No-code data engineering

Automated data transformation

Enterprise-grade LLM

DataSteroid