Benchmarking API for ITS’s AI Algorithms

Overview

In the dynamic space of financial trading, artificial intelligence has become a useful tool for predicting future trends in the stock market. Human traders’ techniques driven by emotion and intuition have shown limitations in the changing market. Other techniques like Deep Neural Networks fail because there is not a sufficiently large and representative dataset to cover all circumstances and anomalies in price chart data.​ Instead Intelletic’s current trading system uses Nementa’s cortical learning algorithm called NuPIC. This algorithm allows human-like learning without the necessity of pre-defined training sets. However this algorithm is built on Python 2.7, which cannot save models greater than 2 GB. Intelletic also requires an API for benchmarking algorithms that may be used in the future.​ The goals of our project are to translate NuPIC into Python 3.9+ and create a benchmarking algorithm to test the performance and accuracy of these algorithms.​​

Objectives

The architecture of our system can be conceptually likened to it in terms of how the libraries interact.

“Algorithm Execution Engine” for running algorithms, “Data & Model Management” for handling data and models, “Results Processor” for evaluating performance, “File Modifier/Processing” for file preparation, “Dataset Loader” for loading data, “Output Processing” for processing results, and “Chart Generator” for creating charts.

Approach

The practical applications of the “Benchmarking API for ITS’s AI Algorithms” project are multifaceted.

Firstly, the system is capable of evaluating and comparing the accuracy and efficiency of various unsupervised AI algorithms. This functionality is crucial for meeting customers’ demands for scalability and performance, as our goal is to assess the effectiveness of various AI algorithms. By benchmarking different APIs against NuPIC’s HTM algorithm, we will gain a more comprehensive understanding of the performance of these different algorithms, thereby better planning for the future, whether it is considering the need to switch to more powerful algorithms, or possibly improving the HTM algorithm (if necessary).

As for organizational impact, our project to create benchmarking APIs and update the software used by Intelletic will help them enhance the predictive accuracy and speed of their algorithms. This will provide better and faster results for end-users. Clearly, this is also beneficial to Intelletic as it will help them maintain the modernity of their software while improving their products and predictive capabilities, which in turn will promote company growth.

The use of the benchmarking tool and end-users depend on API calls. These API calls can be used to perform a variety of tasks, such as benchmarking existing or new AI algorithms, updating or adding new datasets to improve our results, or generating chart information about algorithm performance.

At a broader level, the project will impact the financial sector and its various influenced areas, although this impact may not be very intuitive. However, the focus on benchmarking and developing AI algorithms will help further advance the field of AI. Since Intelletic products are primarily aimed at organizations and investment groups, any organization using Intelletic will receive automated and accurate stock market forecasts for the futures market.

Outcomes

Upon correctly entering the model parameter path and the corresponding dataset path, the API will execute model predictions according to the model configuration file. The model execution progress will be displayed on the command line, showing both the total time required for model execution and the current running efficiency.

Right: After completing model predictions, all prediction results will be visualized using the matplotlib library. Users can analyze predictive performance based on the visualized results.

Left: The final MSE result of the HTMPrediction model in one-step time series prediction is 121.56, and in five-step time series prediction is 640.38.

Right: The curve in the figure represents the change in MSE loss for different time spans (one-step and five-step) under different iteration steps (larger iteration steps represent longer time periods involved in the calculation).