Modernizing Data: The Journey from Clutter to Clarity for a Corporate Information Platform

Financial news/aggregator

Modernizing Data: The Journey from Clutter to Clarity for a Corporate Information Platform

02 Engineers

03 Months

60%

Increase in Data Accuracy

Technology

Python, Django, CoreNLP, NLTK, Beautifulsoup, Selenium, Pandas

Services

Rapid Application Development

Location

Colorado

Overview

A Colorado-based information aggregation platform provided information related to companies and company directors listed on the Russell 3000 Index using U.S. Securities and Exchange Commission (SEC) data. They reached out to us for a revamp of their existing platform. The objective was to build a user-friendly, real-time information aggregation platform to make data readily accessible, interpretable, and shareable for their users, thus enhancing their overall user experience.

Challenges

User-Unfriendly Platform: The existing platform was replete with legal jargon, lacked organization, and was not user-friendly.
Time-Consuming Data Sorting: Users had to browse multiple sites to gather all the required information, a task too time-consuming for busy senior executives.
Limited Shareability: The existing platform didn't allow users to share reviewed information with their peers via email.
Lack of Premium Content: Users couldn't create valuable business collateral, such as white papers, within the existing system.

High Level Design Architecture

Solutions

Upon choosing TechVariable for its technical expertise, offshore development capabilities, and quick turn-around, the project’s initial phase was completed in three months, after which the company extended its engagement for further enhancements. We undertook the following steps:

Designed a Scalable, Modular Platform: We developed the platform with future scalability in mind, allowing for the expansion of features as the business grows.
Leveraged Microservices Architecture: We used the Django Rest Framework as a back-end web framework, PostgreSQL for DB, and React JS for the front end.
Implemented NLP Algorithms: Natural Language Processing algorithms were used to extract meaningful entities from unstructured public data using a Spacy custom pipeline.
Introduced Relevance Scoring: Documents were scored based on semantic analysis and word embedding to identify high-relevance documents.
Utilized Third-Party Services: Real-time information such as stock prices was obtained using services like iexcloud.
Improved Search Mechanism: Elasticsearch was used to optimize the searching mechanism, reducing the load on the DB and improving search results.

Result

Enhanced Searchability: Elasticsearch improved the ease of indexing and made resources readily searchable.
Improved Shareability: The platform allows users to share resources internally and externally via shareable links.
Streamlined Data Visualization: The use of NLP to extract data from unstructured formats and display it as a report enabled users to comprehend the data at a glance visually.