As others noted, anomaly detection in time series is a huge topic.

Some specific cases are trivial, such as simple thresholds or triggers based on deviation from a norm. Ultimately though, on non-trivial dynamic processes, the definition of an anomaly is very context-dependent.

There are very different algorithms out there, but the basic idea (with wildly different implementations) is that an algorithm would try to predict the next value(s) on the basis of the past, observe the actual value, and measure how far off the prediction was: if it’s very different, it is considered an anomaly. The problem is to choose a prediction model that is sophisticated enough to take into account all known effects: seasonality, patterns, range of frequencies where random oscillations are expected, self-correlation, measurable inputs that affect your system, etc.

The more known effects you add to your prediction, the more your solution will become a custom dynamic model, departing from off-the-shelf algorithms.

It is possible to use ML to learn patterns vs anomalies, but often not easy, due to the fact that anomalies are by definition rare, and do not necessarily look the same. Without proper care to avoid overfitting, if anomalies are rare enough, an ML algorithm could achieve seemingly good results by just labeling everything as “not anomaly”, which is obviously useless. You will therefore need to assess how expensive a false positive is for your case, vs. a false negative.

I know this is not a specific suggestion for an algorithm, and might not be too helpful, but I’d recommend to:

• Reason on what you know about the dynamics of your system. Seasonality, frequency range, measurable inputs that can affect your system and might feed into the model.
• Assess the cost of false positives vs. false negatives. If false positives are cheap, you can make your algorithm more sensitive, in the attempt to catch all anomalies. Otherwise things get harder. In any case, you’ll need to decide on this important trade off.
• How fast should your alerting system react? If it needs to be very fast, it won’t be able to wait and weight in more observations to see if a seemingly strange pattern normalizes (or, equivalently, use longer windows for weighted averages). This makes your alerting more susceptible to noise, and might put limits on the false positive rate.
• Start simple, evaluate, and add up: how many anomalies do you catch with a simple threshold model, maybe normalizing seasonalities? What kind of anomalies are missed? If you add thresholds also on the first derivative (growth, as opposed to absolute value), would such anomalies be detected?

In sum, know your system, and split the big problem into separate ones that can each be solved by a known algorithm.

I believe that ML in this case is not a silver bullet, and comes after a proper problem definition. It’s ok to introduce ML if necessary, but deep knowledge of the system, if possible at all, will likely beat ML in this case. Even if you do use ML, you will need to pick an algorithm, prepare its input, and tweak its parameters based on such knowledge of your system.

Hi !

This is a vast question!

Yes, ML could be a good fit, however the NFL theorem would tell you that there are no easy shortcuts to success and every tips we could give you will have to be put to the test with your own data!

I mostly work on Natural Language Processing problems thus time series are not really my domain of expertise but I can try to give you some clues:

There are a lot of ML tools able to be used for anomaly / outliers / novelty detection such as SVM, Isolation Forest, LOF, Auto Encoders, Spectral Residual, Seq2Seq… and drift detection may require other solutions.

The only (quite simple) TS problem I’ve been confronted has been solved by a Spectral Residual approach for the outlier detection part and by a Context Aware Maximum Mean Discrepancy (CAMMD) approach for the drift detection part thus if I had to solve another similar problem I would probably give those ones a shot.

As a general advice, even if you do not implement your solution with the Python language, the documentation of scikit-learn may give you some basic understanding of some possible approaches:

• 2.7. Novelty and Outlier Detection — scikit-learn 1.9.0 documentation
• Comparing anomaly detection algorithms for outlier detection on toy datasets — scikit-learn 1.9.0 documentation

The documentation of alibi-detect could give you some keywords to look for too: GitHub - SeldonIO/alibi-detect: Algorithms for outlier, adversarial and drift detection · GitHub .

I hope this helps!