Virtual fields

Virtual fields allow you to derive new values from your data in real time, eliminating the need for up-front data structuring, enhancing flexibility and efficiency.

Virtual fields allow you to derive new values from your data in real time. They eliminate the need for up-front planning of how to structure or transform your data. Instead, send your data as-is and then use virtual fields to manipulate your data in real-time during queries. The feature is also known as derived fields, but Axiom’s virtual fields have some unique properties that make them much more powerful.

Virtual fields are query-time representations only. They don’t create new data in storage.

Virtual fields aren’t currently supported for metrics datasets.

Creating a virtual field

To create a virtual field, follow these steps:

1. Go to the Datasets tab.
2. Select the dataset where you want to create the virtual field.
3. Click Virtual fields in the top right. You see a list of all the virtual fields for the dataset.
4. Click Add virtual field.
5. Fill in the following fields:
   • Name and Description help your team understand what the virtual field is about.
   • Expression is the formula applied to every event to calculate the virtual field. The expression produces a result such as a boolean, string, number, or object. The Preview section displays the result of applying the expression to some of your data. Use this section to verify the expression and the resulting values of the virtual field.

The power of virtual fields is in letting you manipulate data on read instead of on write, allowing you to adjust and update virtual fields over time as well as easily add new ones without worrying that the data has already been indexed.

Usage

Visualizations

Virtual fields are available as parameters to visualizations but, as the type of a virtual field can be any of the supported types, it’s important to make sure that you use a virtual field that produces the correct type of argument.

Filters

Virtual fields are available in the filter menu and all filter options are presented. It’s important to ensure that you are using a supported filter operation for the type of result your virtual field produces.

Group By

Virtual fields can be used for segmentation in the same way as any standard field.

Virtual fields vs ingest-time parsing

When deciding how to structure your data, consider the trade-offs between using virtual fields (query-time parsing) and parsing fields at ingest time.

When to use virtual fields

Virtual fields are ideal when:

• You need flexibility to experiment with different data transformations.
• Your query patterns are still evolving and you’re not sure which fields you’ll need.
• You want to derive new fields without re-ingesting historical data.
• The fields are used infrequently or for ad-hoc analysis.

When to parse at ingest time

For high-performance use cases, parsing fields at ingest time is often the better choice:

• Query performance: Virtual fields re-parse data on every query execution. For frequently queried fields, especially during incident response when teams run many ad-hoc queries, this adds latency. Parsed fields at ingest time reduce your storage and query hours usage. For more information, see Virtual fields for simple transformations.
• Compression benefits: When you parse fields from a JSON body to top-level fields, Axiom uses specialized data-aware compression techniques based on the detected data types. This results in more efficient storage compared to keeping data as unparsed strings. For more information, see Overusing runtime JSON parsing.

Storage considerations

Parsing fields from a body to top-level attributes doesn’t double your storage usage. Axiom’s compression is optimized for typed, structured data at the top level, and stores parsed fields efficiently. However, there is some data redundancy if you keep the original body field alongside the parsed fields. To minimize redundancy, remove the parsed fields from the body after extraction during your ingest pipeline.

For a balance between flexibility and performance, parse your most frequently queried fields at ingest time while using virtual fields for less common or experimental transformations.

Reference

Virtual fields are APL expressions and share all the same functions and syntax as APL expressions. For more information, see Introduction to APL.

The list of APL scalar functions:

• String functions
• Math functions
• Array functions
• Conversion functions
• Hash functions
• DateTime/Timespan functions
• Rounding functions
• Conditional functions
• IP functions

Virtual fields may reference other virtual fields. The order of the fields is important. Ensure that the referenced field is specified before the field that references it.