Current research

PhD in Interpretable AI (Symbolic NARMAX)

Symbolic NARMAX systems (arbitrary functions such as Polynomials, Wavelets, RBFs (like GMMs), FIRs, IIRs, etc.) are the main alternative to artificial neural networks (ANNs) for black-box fitting or data (and stochastic process) analysis and processing. For system fitting, they are, however, used essentially for small to mid-size systems (or in situations where interpretability and provable correctness is paramount), since SotA methods are either limited in fitting capabilities (FOrLSR-based methods) or prohibitively slow (genetic algorithms). NARMAX models have many advantages over ANNs such as interpretability, ease of modification and analysis, computational efficiency, and marginal data requirements compared to ANNs (500 or fewer floats vs GB/TB/PB of data). Thus, it's of interest to push the limits of the FOrLSR (Forward Orthogonal Least Squares Regression) framework. This PhD's goal is to make the FOrLSR framework capable of reliably fitting increasingly more complex systems with each paper, such that the more efficient and expressive symbolic NARMAX systems can be used instead of the black-box neural networks in an increased number of applications. The focus is on supporting general symbolic NARMAX expansions to allow users to fit their domain-specific functions and expressions, to customize fitting quality and interpretability.

Paper 1: Arborescent Orthogonal Least Squares Regression (AOrLSR)

The paper constitutes a large jump compared to the SotA, as the presented ML algorithm (linear algebra, graph theory, combinatorial optimization, numerics) beats the most modern methods by a very large margin in speed, precision, stability and model conciseness. It allows to reliably fit large classes of heavily non-linear and dynamic multi-channel systems which were previously deemed too complex to correctly fit (rational, nested inside equations, discontinuous, non-differentiable, acausal, etc). It can also be used for data compression and de-noising (sparse reconstruction) or just as a fancy (over / underdetermined) system solver minimizing the L0-norm (NP-Hard Problem).

The paper contains:

• Translation of the original FOrLSR algorithm into Matrix-form to be used for big data on GPUs/TPUs/NPUs, etc.
• Flattening of the complexity from O(n²) to O(n) by making it recursive
• Restructuring the algorithm into a search-tree for better model search
• 5 proven theorems, 4 corollaries, 1 pruning mechanism allowing to explore the solution space while computing a marginal fraction of it (0.065% - 5% depending on system complexity), with up to 55'000x speed-ups in fitting times (on my old PC, scales with computing power)
• Important ground-work for upcoming follow-up papers in my PhD.
• A benchmark suite of difficult-to-fit systems to better compare NARMAX fitting algorithms

Read or download the AOrLSR paper.
Fully implemented in my python library.

Paper 2: (Symbolic Oscillator)

The paper presents a highly optimized JIT-compiler for generalized rational NARMAX systems, allowing to measure (rather than approximate) the domain-specific user-defined functions during fitting and validation. This allows to further increase the complexity of reliably fitted systems, in addition to allowing their integration in ML-pipelines as compiled python objects.

Paper contains:

• Optimal dispatch between CPU and GPU to run the arbitrary user system (non-linear, recursive, stochastic, etc.) with maximal efficiency.
• Merging repeated subexpressions to avoid redundant recomputations
• Mathematical simplification of expressions (factoring out terms, etc) for efficiency
• Optimal toggling scheme to retrieve samples predating the bufferstart depending on hardware accelerator.
• Further optimizations to speed-up evaluation

Status: Math: 90%. Code: 90% - parts of it already present in my python library
Paper to be submitted this year for publication + full release in library.

Small in-between paper: dERR - ERR metric stabilized for rational NARMAX systems.

This paper proposes an update to the ERR similarity metric to stabilize it against low-variance denominator regressors. This is a long-known problem in fitting rational regressors, that was, however, mathematically misattributed to other phenomena.

Paper contains:

• Discussions of the origin of the problem (and why current estimates are incorrect)
• Demonstrates the origin with a mathematical proof
• Demonstrates the origin with synthetic numeric examples controlling the effect
• Proposes a more refined internal metric that is stable against the problem
• Mathematical proofs that the metric works
• Numerical proofs (aka fitting examples) that it works

Paper to be submitted this year for publication & integration in the library.

Paper 3: Dictionary Morphing Orthogonal Least Squares Regression (DMOrLSR)

Optimization pipeline allowing the framework to adapt the user-provided domain-specific functions to the system (or data or stochastic process) for more precise and smaller models. (Paper 1 only allows to use the user-provided functions as-is, while this allows the library to optimize them further)

This paper (DMOrLSR) morphs the user-given regressor (functions) to allow expansions which adapt their terms to the system output (imagine a fourier transform which finds the exact peaks and only contains those terms, being sparse). This is based on genetic algorithms, linear algebra, matrix calculus / infinitesimal optimization, with closed form gradients and Hessians independent of the user-passed function, number of arguments and data.

Paper contains:

• A Mathematical property and algorithm to determine the morphability of arbitrary user-expressions (is morphing this worth it?)
• Genetic algorithm determining what multiplicative coefficients and supplementary dictionary entries to add (how to morph the expression)
• Closed-Form self-orthogonalizing Gradient and Hessian and many theorems about their behaviors and properties.
• Trust region method optimization of (non-nested) NARMAX regression expressions based on above gradients and Hessian
• local coefficient infinitesimal optimization procedure searching the best coefficients inside the chosen non-linearity
• Infinitesimal optimizer for the entire expression
• Very fancy application examples (Infinite precision Fourier & Wavelet transforms (aka beating the Heisenberg uncertainty), Symbolic Neural Networks, Mixtures of different radial basis functions, etc.)

Already present in my python library but not exposed by current API.
Paper to be submitted this year for publication & full release in library.

Paper 4: Batch Robust Forward Orthogonal Barron Regression (BrFOrBr)

Batching and (outlier/noise) robust fitting.

Paper contains:

• Robust fitting: Least-squares is optimal in the case of gaussian noise but is very sensitive to large outliers (Sudden unplanned events, sensor glitches, etc.), robust fitting makes the fitting stable against such problems (often used in robotics, self-driving cars and other sensor-heavy industries).
• Batching: Generalize the procedures in paper 1 & 3 to be in Tensor-form for heavily parallelized fitting of large data sets (big data).

Status: Math: 50%. Code: 20%
Equivalence to Neural Networks: Data Batching (Fine tuning) + Data-preprocessing (outlier removal and heavy noise compensation built into the fitting algorithm)

Paper 5: Lean Arborescent Search Algorithm for Generalized NARMAX Expressions (LASAGNE)

Merge the algorithms from all the previous papers into the first one to tremendously increase the complexity of reliably fitted systems in addition to providing supplementary sparsity theorems and optimizations to accelerate fitting. This paper also introduces support for auto-regressive MIMO fitting with compiled systems.

Status: Math: 30%. Code: 0%

Paper 6: Multi-layer LASAGNE for Deep NARMAX-Networks

Reinterprets the generalized symbolic NARMAX model as a single-layer network and generalizes it to be multi-layer, with the advantage over traditional artificial neural networks (ANNs) that the network chooses its optimal structure (size, connections and activation functions) by itself, while remaining interpretable (human readable-equations).

Status: Math: 5%. Code: 0%
Equivalence to Neural Networks: Multi-layer Neural Network with build-in architecture selection and Data-cleaning

Python Library

The above algorithms are offered in the soon to-be-released GPU-accelerated open-source python machine learning package.

The library makes optimal sparse least squares fitting of any functions/data passed by the user for the given system response. All provided examples are with severely non-linear auto-regressive systems, however, the user can pass dictionaries with any type of functions/data. This is thus a very general mathematical framework supporting for example wavelet-, RBF, polynomial, FIR/IIR, binary systems, etc. fitting. The papers also contain an example of using the library to fit a nested expansion in |x|^n inside a fraction or of nesting non-linearities inside other non-linearities.

Supplementary NARMAX research

Based on my previously described papers, further ameliorations and papers are planned:

• Together with a Technische Universität Berlin professor, research is planned to replace some layers of neurons in audio-generating Neural Networks (for Text to Speech generation, Music synthesis, etc) by more efficient and interpretable NARMAX systems.
• In depth study of the effect of data-set size and excitation signal on the fitting quality and convergence speed
• Ameliorations to the genetic vector space generator sub-algorithm
• Morphing support for rational NARMAXes and noise stabilization (expanding paper 3)

Dimensionality selection paper

This machine learning algorithm automatically selects the amount of information worth keeping to represent data or a digital system. The applications are for example finding ideal compression ratios (storage and transmission) or ideal approximations of systems.

I propose a version of the algorithm by Zhu and Ghodsi (Alternative Link) in 2006, where the recursion is unrolled to a much more efficient matrix-form algorithm, which also adds the ability to introduce constraints on the dimensionality selection.

Paper contains:

• The mathematical derivation of the optimized algorithm
• C++ and a Python implementation for benchmarks and examples
• Benchmarks comparing this algorithm to the original
• Examples to illustrate the use and the new features

Paper is mostly written, submission date not yet determined (PhD papers have priority).

Contributions at current / former employers

TRACK3D (AI-supervised 3D printing)

My research on custom algorithms and data structures for the 3D printer includes high-throughput GCode analysis and movement extraction, algorithmic pointcloud 3D segmentation, Optical scanner algorithms and data stabilization, etc. sadly stays behind the walls of my NDA.

However, in the context of being a system architect (theoretical informatics research), I did extensive research on modern error-management methods for C++ (Analyzed C, Rust, Bun, Go, etc.) to propose ameliorated language constructs for C++ that allow for advanced, no-throw but value-based error management with compile-time optimizations. This will soon be open-sourced as C++23 library and an accompanying paper.

In addition, in the same system architecture (theoretical informatics research), I designed a rule book of about 60 pages on guidelines for the most modern and safe C++23 that will soon be published. It took inspiration from the C++ coding guidelines of Google, Firefox, CPP Core, CCP-ISO amongst others and supplemented by many tests and benchmarks.

u-he (AI-Audio Generation & Analysis)

My research on ADAA (anti-derivative anti-aliasing for distortions and saturations), AI-based wavetable extraction for Zebra 3, clustering for audio samples, wavelet methods as fourier generalization, psycho-acoustics focussed reverb algorithms, etc. sadly stays behind the walls of my NDA and won't be published, being algorithms used in commercial audio plug-ins.

Eigen contributions

Eigen is a open-source highly optimized lazy-evaluation C++ expression-tree-based linear algebra library, which is used amongst others in Tensorflow, on which Google's AIs/self-driving cars/ game engines / physics simulations / etc. are based.

Contributions:

• Meta-templated DenseBase Concatenation sadly closed due to me not reacting fast enough to their change requests, but a good demonstration of my meta-templating and compile-time optimization skills in C++.
• Meta-templated Binary indexing as found in Numpy/Matlab
• ArgSort + generalized ArgMin/ArgMax. Dispatch ot the most efficient algorithm depending on the task and data-size. Rejected being convenience functions.
• Enables compile-time access to most-used Eigen classes. Accepted and merged.
• Many bug reports (incorrectly handled acessing pattern, poor performance of some operations, etc.). Taken into considerations and fixed.