A Malaysian news agency wanted to automate insight-mining from enormous amounts of data collected from various disparate sources, so they could speed up generation of useful reports on key societal trends. Their reports were often used by the government to understand trends such as drug usage. Currently, data was being captured from social media, newspapers and physical scanning of documents, before being manually stored in spreadsheets and analysed for trends.
The current process was labor-intensive, time consuming and inefficient. It was also prone to errors and was not very scalable.
Technical challenges in existing solution
Manual data entry: Data from different sources was manually entered into spreadsheets.
No standardization: Data came in different formats, sizes and languages, and needed to be harmonized.
Limitations on number of requests handled: Current AWS Lambda services had limitations on the number of requests processed at any given time.
Lack of customization: No access control through role management.
Reports lacked insights: No visualization of data to report high level trends.