All Collections
Case Studies
Global event collection using Machine learning
Global event collection using Machine learning
APEXE3HQ avatar
Written by APEXE3HQ
Updated over a week ago

APEX:E3's partnership with The University of Oxford has allowed research students in computer science and qualitative research to access its efficient API. The study, led by Lloyd Arnold, Sebastian Foulger, Mathew Kurnia, Felipe Nuti, and Maté Soós, used APEX’s API to stress test trading strategies during past significant events in crypto and stock markets using a backtesting framework and machine learning. The study utilised a MongoDB database and Nest.js for the backend and Python for the rest, utilising a database API, NLP API, and multiple data scrapers.

To improve efficiency and scalability, a microservice architecture was employed for the backtesting feature of Apex:E3's partnership with The University of Oxford. The system consisted of a Nest.js-based RESTful API and a MongoDB database. Global events were collected using the GDELT database, which monitored news from various languages and sources. The GDELT project offered three primary modes of access: analysis service, Google BigQuery, and raw data download. This approach allowed the system to better handle increased demand by deploying new scrapers and adding inference APIs.

Two scrapers were developed: a live scraper that downloads the latest daily csv from GDELT, filters, converts entries, and adds events to the database, saving cost. The second is a manually deployable scraper that searches the entire GDELT database using Google BigQuery, converts entries, and ingests into a database. Fake news classification was also considered for storing news articles, although it's not a perfected area yet."

Twitter and CoinmarketCap scrapers were also created. The Twitter scraper uses the official API to obtain tweets, calculates sentiment using an inference API, averages scores, and updates the database, with filtering by keyword and hashtag options. The CoinMarketCap scraper uses their API to get news articles, tags assets mentioned, and stores the timestamp, sentiment, source URL, category, title, and stocks in the database.

In conclusion, the Oxford-APEX partnership led to the development of a risk management solution for real-time backtesting of events. The project included tools such as a Global Data Persister and Collator, using large language models and microservice architecture for scalable data collection in APEX’s backtesting framework.

Did this answer your question?