This paper introduces a flexible Transformer-based model for detecting anomalies in system logs. By embedding log templates with a pre-trained BERT model and incorporating positional and temporal encoding, it captures both semantic and sequential context within log sequences. The approach supports variable sequence lengths and configurable input features, enabling extensive experimentation across datasets. The model performs supervised binary classification to distinguish normal from anomalous patterns, using a [CLS]-like token for sequence-level representation. Overall, it pushes the boundaries of log-based anomaly detection by integrating modern NLP and deep learning techniques into system monitoring.This paper introduces a flexible Transformer-based model for detecting anomalies in system logs. By embedding log templates with a pre-trained BERT model and incorporating positional and temporal encoding, it captures both semantic and sequential context within log sequences. The approach supports variable sequence lengths and configurable input features, enabling extensive experimentation across datasets. The model performs supervised binary classification to distinguish normal from anomalous patterns, using a [CLS]-like token for sequence-level representation. Overall, it pushes the boundaries of log-based anomaly detection by integrating modern NLP and deep learning techniques into system monitoring.

Transformer-Based Anomaly Detection Using Log Sequence Embeddings

Abstract

1 Introduction

2 Background and Related Work

2.1 Different Formulations of the Log-based Anomaly Detection Task

2.2 Supervised v.s. Unsupervised

2.3 Information within Log Data

2.4 Fix-Window Grouping

2.5 Related Works

3 A Configurable Transformer-based Anomaly Detection Approach

3.1 Problem Formulation

3.2 Log Parsing and Log Embedding

3.3 Positional & Temporal Encoding

3.4 Model Structure

3.5 Supervised Binary Classification

4 Experimental Setup

4.1 Datasets

4.2 Evaluation Metrics

4.3 Generating Log Sequences of Varying Lengths

4.4 Implementation Details and Experimental Environment

5 Experimental Results

5.1 RQ1: How does our proposed anomaly detection model perform compared to the baselines?

5.2 RQ2: How much does the sequential and temporal information within log sequences affect anomaly detection?

5.3 RQ3: How much do the different types of information individually contribute to anomaly detection?

6 Discussion

7 Threats to validity

8 Conclusions and References

\

3 A Configurable Transformer-based Anomaly Detection Approach

In this study, we introduce a novel transformer-based method for anomaly detection. The model takes log sequences as inputs to detect anomalies. The model employs a pretrained BERT model to embed log templates, enabling the representation of semantic information within log messages. These embeddings, combined with positional or temporal encoding, are subsequently inputted into the transformer model. The combined information is utilized in the subsequent generation of log sequence-level representations, facilitating the anomaly detection process. We design our model to be flexible: The input features are configurable so that we can use or conduct experiments with different feature combinations of the log data. Additionally, the model is designed and trained to handle input log sequences of varying lengths. In this section, we introduce our problem formulation and the detailed design of our method.

\ 3.1 Problem Formulation

We follow the previous works [1] to formulate the task as a binary classification task, in which we train our proposed model to classify log sequences into anomalies and normal ones in a supervised way. For the samples used in the training and evaluation of the model, we utilize a flexible grouping approach to generate log sequences of varying lengths. The details are introduced in Section 4

\ 3.2 Log Parsing and Log Embedding

In our work, we transform log events into numerical vectors by encoding log templates with a pre-trained language model. To obtain the log templates, we adopt the Drain parser [24], which is widely used and has good parsing performance on most of the public datasets [4]. We use a pre-trained sentence-bert model [25] (i.e., all-MiniLML6-v2 [26]) to embed the log templates generated by the log parsing process. The pre-trained model is trained with a contrastive learning objective and achieves state-ofthe-art performance on various NLP tasks. We utilize this pre-trained model to create a representation that captures semantic information of log messages and illustrates the similarity between log templates for the downstream anomaly detection model. The output dimension of the model is 384.

\ 3.3 Positional & Temporal Encoding

The original transformer model [27] adopts a positional encoding to enable the model to make use of the order of the input sequence. As the model contains no recurrence and no convolution, the models will be agnostic to the log sequence without the positional encoding. While some studies suggest that transformer models without explicit positional encoding remain competitive with standard models when dealing with sequential data [28, 29], it is important to note that any permutation of the input sequence will produce the same internal state of the model. As sequential information or temporal information may be important indicators for anomalies within log sequences, previous works that are based on transformer models utilize the standard positional encoding to inject the order of log events or templates in the sequence [11, 12, 21], aiming to detect anomalies associated with the wrong execution order. However, we noticed that in a common-used replication implementation of a transformer-based method [5], the positional encoding was, in fact, omitted. To the best of our knowledge, no existing work has encoded the temporal information based on the timestamps of logs for their anomaly detection method. The effectiveness of utilizing sequential or temporal information in the anomaly detection task is unclear.

\ In our proposed method, we attempt to incorporate sequential and temporal encoding into the transformer model and explore the importance of sequential and temporal information for anomaly detection. Specifically, our proposed method has different variants utilizing the following sequential or temporal encoding techniques. The encoding is then added to the log representation, which serves as the input to the transformer structure.

\

3.3.1 Relative Time Elapse Encoding (RTEE)

We propose this temporal encoding method, RTEE, which simply substitutes the position index in positional encoding with the timing of each log event. We first calculate the time elapse according to the timestamps of log events in the log sequence. Instead of using the log event sequence index as the position to sinusoidal and cosinusoidal equations, we use the relative time elapse to the first log event in the log sequence to substitute the position index. Table 1 shows an example of time intervals in a log sequence. In the example, we have a log sequence containing 7 events with a time span of 7 seconds. The elapsed time from the first event to each event in the sequence is utilized to calculate the time encoding for the corresponding events. Similar to positional encoding, the encoding is calculated with the above-mentioned equations 1, and the encoding will not update during the training process.

\

3.4 Model Structure

The transformer is a neural network architecture that relies on the self-attention mechanism to capture the relationship between input elements in a sequence. The transformer-based models and frameworks have been used in the anomaly detection task by many previous works [6, 11, 12, 21]. Inspired by the previous works, we use a transformer encoder-based model for anomaly detection. We design our approach to accept log sequences of varying lengths and generate sequence-level representations. To achieve this, we have employed some specific tokens in the input log sequence for the model to generate sequence representation and identify the padded tokens and the end of the log sequence, drawing inspiration from the design of the BERT model [31]. In the input log sequence, we used the following tokens: is placed at the start of each sequence to allow the model to generate aggregated information for the entire sequence, is added at the end of the sequence to signify its completion, is used to mark the masked tokens under the self-supervised training paradigm, and is used for padded tokens. The embeddings for these special tokens are generated randomly based on the dimension of the log representation used. An example is shown in Figure 1, the time elapsed for , and are set to -1. The log event-level representation and positional or temporal embedding are summed as the input feature of the transformer structure.

\ 3.5 Supervised Binary Classification Under this training objective, we utilize the output of the first token of the transformer model while ignoring the outputs of the other tokens. This output of the first token is designed to aggregate the information of the whole input log sequence, similar to the token of the BERT model, which provides an aggregated representation of the token sequence. Therefore, we consider the output of this token as a sequence-level representation. We train the model with a binary classification objective (i.e., Binary Cross Entropy Loss) with this representation.

\

:::info Authors:

  1. Xingfang Wu
  2. Heng Li
  3. Foutse Khomh

:::

:::info This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license.

:::

\

Market Opportunity
Bert Logo
Bert Price(BERT)
$0.020685
$0.020685$0.020685
+2.17%
USD
Bert (BERT) Live Price Chart
Disclaimer: The articles reposted on this site are sourced from public platforms and are provided for informational purposes only. They do not necessarily reflect the views of MEXC. All rights remain with the original authors. If you believe any content infringes on third-party rights, please contact service@support.mexc.com for removal. MEXC makes no guarantees regarding the accuracy, completeness, or timeliness of the content and is not responsible for any actions taken based on the information provided. The content does not constitute financial, legal, or other professional advice, nor should it be considered a recommendation or endorsement by MEXC.

You May Also Like

Will XRP Price Increase In September 2025?

Will XRP Price Increase In September 2025?

Ripple XRP is a cryptocurrency that primarily focuses on building a decentralised payments network to facilitate low-cost and cross-border transactions. It’s a native digital currency of the Ripple network, which works as a blockchain called the XRP Ledger (XRPL). It utilised a shared, distributed ledger to track account balances and transactions. What Do XRP Charts Reveal? […]
Share
Tronweekly2025/09/18 00:00
Ripple IPO Back in Spotlight as Valuation Hits $50B

Ripple IPO Back in Spotlight as Valuation Hits $50B

The post Ripple IPO Back in Spotlight as Valuation Hits $50B appeared first on Coinpedia Fintech News Ripple, the blockchain payments company behind XRP, is once
Share
CoinPedia2025/12/27 14:24
Lovable AI’s Astonishing Rise: Anton Osika Reveals Startup Secrets at Bitcoin World Disrupt 2025

Lovable AI’s Astonishing Rise: Anton Osika Reveals Startup Secrets at Bitcoin World Disrupt 2025

BitcoinWorld Lovable AI’s Astonishing Rise: Anton Osika Reveals Startup Secrets at Bitcoin World Disrupt 2025 Are you ready to witness a phenomenon? The world of technology is abuzz with the incredible rise of Lovable AI, a startup that’s not just breaking records but rewriting the rulebook for rapid growth. Imagine creating powerful apps and websites just by speaking to an AI – that’s the magic Lovable brings to the masses. This groundbreaking approach has propelled the company into the spotlight, making it one of the fastest-growing software firms in history. And now, the visionary behind this sensation, co-founder and CEO Anton Osika, is set to share his invaluable insights on the Disrupt Stage at the highly anticipated Bitcoin World Disrupt 2025. If you’re a founder, investor, or tech enthusiast eager to understand the future of innovation, this is an event you cannot afford to miss. Lovable AI’s Meteoric Ascent: Redefining Software Creation In an era where digital transformation is paramount, Lovable AI has emerged as a true game-changer. Its core premise is deceptively simple yet profoundly impactful: democratize software creation. By enabling anyone to build applications and websites through intuitive AI conversations, Lovable is empowering the vast majority of individuals who lack coding skills to transform their ideas into tangible digital products. This mission has resonated globally, leading to unprecedented momentum. The numbers speak for themselves: Achieved an astonishing $100 million Annual Recurring Revenue (ARR) in less than a year. Successfully raised a $200 million Series A funding round, valuing the company at $1.8 billion, led by industry giant Accel. Is currently fielding unsolicited investor offers, pushing its valuation towards an incredible $4 billion. As industry reports suggest, investors are unequivocally “loving Lovable,” and it’s clear why. This isn’t just about impressive financial metrics; it’s about a company that has tapped into a fundamental need, offering a solution that is both innovative and accessible. The rapid scaling of Lovable AI provides a compelling case study for any entrepreneur aiming for similar exponential growth. The Visionary Behind the Hype: Anton Osika’s Journey to Innovation Every groundbreaking company has a driving force, and for Lovable, that force is co-founder and CEO Anton Osika. His journey is as fascinating as his company’s success. A physicist by training, Osika previously contributed to the cutting-edge research at CERN, the European Organization for Nuclear Research. This deep technical background, combined with his entrepreneurial spirit, has been instrumental in Lovable’s rapid ascent. Before Lovable, he honed his skills as a co-founder of Depict.ai and a Founding Engineer at Sana. Based in Stockholm, Osika has masterfully steered Lovable from a nascent idea to a global phenomenon in record time. His leadership embodies a unique blend of profound technical understanding and a keen, consumer-first vision. At Bitcoin World Disrupt 2025, attendees will have the rare opportunity to hear directly from Osika about what it truly takes to build a brand that not only scales at an incredible pace in a fiercely competitive market but also adeptly manages the intense cultural conversations that inevitably accompany such swift and significant success. His insights will be crucial for anyone looking to understand the dynamics of high-growth tech leadership. Unpacking Consumer Tech Innovation at Bitcoin World Disrupt 2025 The 20th anniversary of Bitcoin World is set to be marked by a truly special event: Bitcoin World Disrupt 2025. From October 27–29, Moscone West in San Francisco will transform into the epicenter of innovation, gathering over 10,000 founders, investors, and tech leaders. It’s the ideal platform to explore the future of consumer tech innovation, and Anton Osika’s presence on the Disrupt Stage is a highlight. His session will delve into how Lovable is not just participating in but actively shaping the next wave of consumer-facing technologies. Why is this session particularly relevant for those interested in the future of consumer experiences? Osika’s discussion will go beyond the superficial, offering a deep dive into the strategies that have allowed Lovable to carve out a unique category in a market long thought to be saturated. Attendees will gain a front-row seat to understanding how to identify unmet consumer needs, leverage advanced AI to meet those needs, and build a product that captivates users globally. The event itself promises a rich tapestry of ideas and networking opportunities: For Founders: Sharpen your pitch and connect with potential investors. For Investors: Discover the next breakout startup poised for massive growth. For Innovators: Claim your spot at the forefront of technological advancements. The insights shared regarding consumer tech innovation at this event will be invaluable for anyone looking to navigate the complexities and capitalize on the opportunities within this dynamic sector. Mastering Startup Growth Strategies: A Blueprint for the Future Lovable’s journey isn’t just another startup success story; it’s a meticulously crafted blueprint for effective startup growth strategies in the modern era. Anton Osika’s experience offers a rare glimpse into the practicalities of scaling a business at breakneck speed while maintaining product integrity and managing external pressures. For entrepreneurs and aspiring tech leaders, his talk will serve as a masterclass in several critical areas: Strategy Focus Key Takeaways from Lovable’s Journey Rapid Scaling How to build infrastructure and teams that support exponential user and revenue growth without compromising quality. Product-Market Fit Identifying a significant, underserved market (the 99% who can’t code) and developing a truly innovative solution (AI-powered app creation). Investor Relations Balancing intense investor interest and pressure with a steadfast focus on product development and long-term vision. Category Creation Carving out an entirely new niche by democratizing complex technologies, rather than competing in existing crowded markets. Understanding these startup growth strategies is essential for anyone aiming to build a resilient and impactful consumer experience. Osika’s session will provide actionable insights into how to replicate elements of Lovable’s success, offering guidance on navigating challenges from product development to market penetration and investor management. Conclusion: Seize the Future of Tech The story of Lovable, under the astute leadership of Anton Osika, is a testament to the power of innovative ideas meeting flawless execution. Their remarkable journey from concept to a multi-billion-dollar valuation in record time is a compelling narrative for anyone interested in the future of technology. By democratizing software creation through Lovable AI, they are not just building a company; they are fostering a new generation of creators. His appearance at Bitcoin World Disrupt 2025 is an unmissable opportunity to gain direct insights from a leader who is truly shaping the landscape of consumer tech innovation. Don’t miss this chance to learn about cutting-edge startup growth strategies and secure your front-row seat to the future. Register now and save up to $668 before Regular Bird rates end on September 26. To learn more about the latest AI market trends, explore our article on key developments shaping AI features. This post Lovable AI’s Astonishing Rise: Anton Osika Reveals Startup Secrets at Bitcoin World Disrupt 2025 first appeared on BitcoinWorld.
Share
Coinstats2025/09/17 23:40