Flagging¶

The good, the bad, and the ugly - flexible, reversible flag annotations for every data point.

Why use Time-Stream?¶

Data values on their own rarely tells the whole story - readings go missing, sensors drift, values get corrected. Data analysts often need to know those details. Time-Stream lets you record every one of those caveats inline with the data itself, using flag systems you define, in a form compact enough to carry multiple annotations per row without losing detail.

Simple example¶

Define your flags, link them to a column, and apply with a single call:

core_flags = ["UNCHECKED", "MISSING", "SUSPECT", "CORRECTED"]

# Register the flag system into the TimeFrame
tf.register_flag_system("CORE_FLAGS", core_flags, flag_type="categorical")

# Initialise a new flag column tied to the CORE_FLAGS system
tf.init_flag_column("CORE_FLAGS", "temperature_flags")

# Flag rows where the temperature exceeds 25 as SUSPECT
tf.add_flag("temperature_flags", "SUSPECT", pl.col("temperature") > 25)

shape: (10, 3)
┌─────────────────────┬─────────────┬───────────────────┐
│ time                ┆ temperature ┆ temperature_flags │
│ ---                 ┆ ---         ┆ ---               │
│ datetime[μs]        ┆ f64         ┆ str               │
╞═════════════════════╪═════════════╪═══════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ null              │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ null              │
│ 2023-01-03 00:00:00 ┆ null        ┆ null              │
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ SUSPECT           │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ null              │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ SUSPECT           │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ SUSPECT           │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ SUSPECT           │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ SUSPECT           │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ SUSPECT           │
└─────────────────────┴─────────────┴───────────────────┘

A few lines to enrich the data: “I want to flag my temperature data as SUSPECT when values are greater than 25.”

Key benefits¶

You define the vocabulary. Flag names, values, and meanings are yours to choose.
Scalable. Multiple flag systems can coexist on the same TimeFrame.
Flexible. Choose bitwise, categorical single, or categorical list flag systems to match your data.
Reversible. Flags can be removed as easily as they are added, using the same expression-based interface.

In more detail¶

Flagging in Time-Stream is built around two concepts: Flag systems and Flag columns.

Flag systems¶

A flag system is a user-defined set of named flags. Each flag has a name (a human-readable label used as a descriptive lookup) and a value (the compact representation stored in the DataFrame). Time-Stream supports three flag system types, each suited to a different annotation pattern. You pick the type via the flag_type argument of register_flag_system(), with "bitwise" as the default. Multiple flag systems can coexist on the same TimeFrame - for example a QC_FLAGS bitwise system alongside a PROVENANCE categorical list system.

`bitwise`¶

What it does: Stores flags as powers-of-two integers. Multiple flags combine on a single integer per row via bitwise OR, so any combination of flags can be recorded without losing detail. See Bitwise values for the maths.

When to use: The default choice. Reach for this when a row may legitimately carry several compatible flags at once (e.g. MISSING + INFILLED) and you want compact storage.

Accepted inputs to register_flag_system:
None - produces a default system with a single FLAGGED flag at value 1:
# Default bitwise system - a single FLAGGED flag at value 1
tf.register_flag_system("DEFAULT")
<DEFAULT (FLAGGED=1)>
list[str] - names are sorted and assigned powers of two automatically:
# Pass a list of names and let Time-Stream assign powers of two
tf.register_flag_system("QC_FLAGS", ["OUT_OF_RANGE", "SPIKE", "FLATLINE", "ERROR_CODE"])
<QC_FLAGS (ERROR_CODE=1, FLATLINE=2, OUT_OF_RANGE=4, SPIKE=8)>
dict[str, int] - explicit mapping. Values must be powers of two and unique:
core_flags = {
    "UNCHECKED": 1,
    "MISSING": 2,
    "SUSPECT": 4,
    "CORRECTED": 8,
    "REMOVED": 16,
    "INFILLED": 32,
}
tf.register_flag_system("CORE_FLAGS", core_flags)
<CORE_FLAGS (UNCHECKED=1, MISSING=2, SUSPECT=4, CORRECTED=8, REMOVED=16, INFILLED=32)>
Note

Each value in a bitwise system must be a power of two (1, 2, 4, 8, 16, …) and unique within that system. Time-Stream raises an error if you try to register an invalid bitwise system.

`categorical`¶

What it does: Each row holds exactly one value, or null. Values can be any int or str. Setting a new flag replaces the previous value on each matching row, so flags are mutually exclusive.

When to use: For “one verdict per row” annotations - e.g. an overall QC rating of good, questionable or bad, or a sensor status code column.

Accepted inputs to register_flag_system:
list[str] with flag_type="categorical" - each name is used as both the name and the value:
# A list of names, each used as both the key and the value
tf.register_flag_system("QC", ["good", "questionable", "bad"], flag_type="categorical")
<QC (bad=bad, good=good, questionable=questionable)>
dict[str, int] with flag_type="categorical" - arbitrary integer values:
# A categorical single system - each row holds exactly one value
qc = {"good": 0, "questionable": 1, "bad": 2}
tf.register_flag_system("QC", qc, flag_type="categorical")
<QC (good=0, questionable=1, bad=2)>
dict[str, str] - string-valued dicts are inferred as categorical automatically, so flag_type can be omitted:
# String values imply categorical automatically
codes = {"good": "G", "questionable": "Q", "bad": "B"}
tf.register_flag_system("CODES", codes)
<CODES (good=G, questionable=Q, bad=B)>

`categorical_list`¶

What it does: Defines the same vocabulary as categorical, but applied to a flag column each row holds a list of values rather than a single one. Adding a flag appends to the list, so a row can carry multiple flags simultaneously.

When to use: When you need multiple flags per row and your values are strings or integer codes - for example, a list of provenance labels (“QC’ED”, “INTERPOLATED”) or numeric status codes (100, 404).

Accepted inputs to register_flag_system: The same forms as categorical above (dict[str, int], dict[str, str], or list[str]), but with flag_type="categorical_list". The only difference is when this flag system is applied to a flag column, a categorical_list system will allow multiple flags per row.

Note

categorical_list and bitwise both allow multiple flags to coexist on a single row. The difference is in how they are stored: categorical_list keeps each flag value as a distinct entry in a list column, making the raw data immediately readable; bitwise encodes all active flags into a single integer, which is more compact but requires decoding to interpret. Either type supports add_flag(), remove_flag(), and filter_by_flag().

Flag columns¶

A flag column is a column in the TimeFrame that has been registered against a specific flag system. They are where flags are stored for a given row in your data. Flag columns are what add_flag(), remove_flag(), filter_by_flag() methods operate on.

There are two ways to create a flag column:

1. Initialise a new flag column using init_flag_column().

This creates a new column in the DataFrame, populated with a sensible default, and registers it against the given flag system. The data type of the resulting column depends on the flag system type:

Bitwise: Int64, initialised to 0.
Categorical single (int values): Int32, initialised to null.
Categorical single (string values): Utf8, initialised to null.
Categorical list: List of the underlying value type, initialised to an empty list.

You can also pre-populate the column with a default flag value:

# Pre-populate every row with the UNCHECKED flag (value 1)
tf.init_flag_column("CORE_FLAGS", column_name="temperature_flags", data=1)

shape: (10, 3)
┌─────────────────────┬─────────────┬───────────────────┐
│ time                ┆ temperature ┆ temperature_flags │
│ ---                 ┆ ---         ┆ ---               │
│ datetime[μs]        ┆ f64         ┆ i64               │
╞═════════════════════╪═════════════╪═══════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ 1                 │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ 1                 │
│ 2023-01-03 00:00:00 ┆ null        ┆ 1                 │
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ 1                 │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ 1                 │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ 1                 │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ 1                 │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ 1                 │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ 1                 │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ 1                 │
└─────────────────────┴─────────────┴───────────────────┘

Note

Normally, you would supply a sensible column_name so that you can keep track of your flags. However, if you do omit it, the column is given a default name of __flag__{flag_system_name}, with an integer suffix appended if a name collides.

tf.init_flag_column("CORE_FLAGS")

['__flag__CORE_FLAGS']

2. Register an existing column using register_flag_column().

If you already have a column of the right data type containing valid flag values, you can register it without modifying the data:

tf.register_flag_column("temperature_flags", "CORE_FLAGS")

Time-Stream validates that every non-null value in the column is a known flag value for the given flag system, and raises an error if any are not recognised.

Adding flags¶

Use add_flag() to set a flag on rows that match a given condition. The operation that this method does depends on the flag system type:

Bitwise - the flag is applied with a bitwise OR, so any existing flags on the row are preserved.
Categorical single - the flag replaces the current value on each matching row. Pass overwrite=False to update only rows whose current value is null.
Categorical list - the flag is appended to the list on each matching row. Duplicate appends are a no-op.

Bitwise workflow:

tf.register_flag_system("CORE_FLAGS", {"UNCHECKED": 1, "MISSING": 2, "SUSPECT": 4})
tf.init_flag_column("CORE_FLAGS", "temperature_flags")

tf.add_flag("temperature_flags", "MISSING", pl.col("temperature").is_null())
tf.add_flag("temperature_flags", "SUSPECT", pl.col("temperature") > 25)
tf.add_flag("temperature_flags", "UNCHECKED", pl.col("temperature") < 25)

shape: (10, 3)
┌─────────────────────┬─────────────┬───────────────────┐
│ time                ┆ temperature ┆ temperature_flags │
│ ---                 ┆ ---         ┆ ---               │
│ datetime[μs]        ┆ f64         ┆ i64               │
╞═════════════════════╪═════════════╪═══════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ 1                 │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ 1                 │
│ 2023-01-03 00:00:00 ┆ null        ┆ 2                 │
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ 4                 │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ 1                 │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ 4                 │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ 4                 │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ 4                 │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ 4                 │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ 4                 │
└─────────────────────┴─────────────┴───────────────────┘

Categorical single workflow:

tf.register_flag_system("QC", {"good": 0, "questionable": 1, "bad": 2}, flag_type="categorical")
tf.init_flag_column("QC", "temperature_qc")

# Each row carries exactly one value; later add_flag calls replace the previous value
tf.add_flag("temperature_qc", "good")
tf.add_flag("temperature_qc", "questionable", pl.col("temperature") > 25)
tf.add_flag("temperature_qc", "bad", pl.col("temperature") > 30)

shape: (10, 3)
┌─────────────────────┬─────────────┬────────────────┐
│ time                ┆ temperature ┆ temperature_qc │
│ ---                 ┆ ---         ┆ ---            │
│ datetime[μs]        ┆ f64         ┆ i32            │
╞═════════════════════╪═════════════╪════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ 0              │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ 0              │
│ 2023-01-03 00:00:00 ┆ null        ┆ 0              │
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ 1              │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ 0              │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ 1              │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ 1              │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ 2              │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ 2              │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ 1              │
└─────────────────────┴─────────────┴────────────────┘

Each add_flag call overwrites the previous verdict on matching rows, so the order of calls matters. Use overwrite=False to fill in only the rows that have no verdict yet:

# overwrite=False leaves rows that already have a value untouched
tf.add_flag("temperature_qc", "good", overwrite=False)

shape: (10, 3)
┌─────────────────────┬─────────────┬────────────────┐
│ time                ┆ temperature ┆ temperature_qc │
│ ---                 ┆ ---         ┆ ---            │
│ datetime[μs]        ┆ f64         ┆ i32            │
╞═════════════════════╪═════════════╪════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ 0              │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ 0              │
│ 2023-01-03 00:00:00 ┆ null        ┆ 0              │
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ 2              │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ 0              │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ 2              │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ 2              │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ 2              │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ 2              │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ 2              │
└─────────────────────┴─────────────┴────────────────┘

Categorical list workflow:

tf.register_flag_system(
    "ORIGIN",
    ["API", "USER_INPUT", "MODEL_OUTPUT"],
    flag_type="categorical_list",
)
tf.init_flag_column("ORIGIN", "temperature_origin")

# Append values to each row's list - flags coexist
tf.add_flag("temperature_origin", "API")
tf.add_flag("temperature_origin", "USER_INPUT", pl.col("temperature") > 25)

shape: (10, 3)
┌─────────────────────┬─────────────┬───────────────────────┐
│ time                ┆ temperature ┆ temperature_origin    │
│ ---                 ┆ ---         ┆ ---                   │
│ datetime[μs]        ┆ f64         ┆ list[str]             │
╞═════════════════════╪═════════════╪═══════════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ ["API"]               │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ ["API"]               │
│ 2023-01-03 00:00:00 ┆ null        ┆ ["API"]               │
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ ["API", "USER_INPUT"] │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ ["API"]               │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ ["API", "USER_INPUT"] │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ ["API", "USER_INPUT"] │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ ["API", "USER_INPUT"] │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ ["API", "USER_INPUT"] │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ ["API", "USER_INPUT"] │
└─────────────────────┴─────────────┴───────────────────────┘

The flag_value argument accepts either the flag name as a string or its underlying value (integer or string, depending on the system). The expr argument is any valid Polars expression that returns a boolean Series, or a boolean Series directly (which is useful in conjunction with the boolean Series output from Quality Control). If omitted, the flag is applied to all rows.

Removing flags¶

Use remove_flag() to clear a flag from rows that match a given condition.

Bitwise - clears the bit with a bitwise AND NOT; other flags on the row are untouched.
Categorical single - sets the column value to null on matching rows.
Categorical list - removes all occurrences of the given value from the list on matching rows.

# Bitwise: clear SUSPECT from corrected rows
tf.remove_flag("temperature_flags", "SUSPECT", pl.col(tf.time_name) < correction_date)

# Categorical single: null out the QC verdict on a subset of rows
tf.remove_flag("temperature_qc", "bad", pl.col("temperature") < 20)

# Categorical list: drop a value from each matching row's list
tf.remove_flag("temperature_origin", "USER_INPUT")

As with add_flag(), flag_value can be a name or underlying value, and expr is optional (defaults to all rows).

Note

Removing a flag that is not set on a row has no effect - the operation is safe to call even when the flag is absent.

Filtering by flag¶

Use filter_by_flag() to return a new TimeFrame containing only the rows that carry (or don’t carry) given flags.

Keep only matching rows:

# Keep only rows flagged as SUSPECT
tf_suspect = tf.filter_by_flag("temperature_flags", "SUSPECT")

shape: (6, 3)
┌─────────────────────┬─────────────┬───────────────────┐
│ time                ┆ temperature ┆ temperature_flags │
│ ---                 ┆ ---         ┆ ---               │
│ datetime[μs]        ┆ f64         ┆ i64               │
╞═════════════════════╪═════════════╪═══════════════════╡
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ 4                 │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ 4                 │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ 4                 │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ 4                 │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ 4                 │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ 4                 │
└─────────────────────┴─────────────┴───────────────────┘

Drop matching rows by passing include=False and a list of flags:

# Drop rows flagged as MISSING or SUSPECT
tf_clean = tf.filter_by_flag("temperature_flags", ["MISSING", "SUSPECT"], include=False)

shape: (3, 3)
┌─────────────────────┬─────────────┬───────────────────┐
│ time                ┆ temperature ┆ temperature_flags │
│ ---                 ┆ ---         ┆ ---               │
│ datetime[μs]        ┆ f64         ┆ i64               │
╞═════════════════════╪═════════════╪═══════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ 0                 │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ 0                 │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ 0                 │
└─────────────────────┴─────────────┴───────────────────┘

For bitwise columns, “matching” means any of the requested flag bits are set. For categorical columns, it means the row’s value (or any element of the list, in list mode) is any of the requested flag values.

Decoding and encoding flag columns¶

Raw flag columns are compact but not particularly human-readable. Use decode_flag_column() to replace the raw values with their flag names:

Bitwise - the integer column becomes a List(String) column of active flag names (sorted by ascending bit value). A value of 0 becomes an empty list.
Categorical single - each raw value becomes its flag name.
Categorical list - each value in each list becomes its flag name.

# Replace the raw integer flag column with human-readable flag names
tf_decoded = tf.decode_flag_column("temperature_flags")

shape: (10, 3)
┌─────────────────────┬─────────────┬──────────────────────────┐
│ time                ┆ temperature ┆ temperature_flags        │
│ ---                 ┆ ---         ┆ ---                      │
│ datetime[μs]        ┆ f64         ┆ list[str]                │
╞═════════════════════╪═════════════╪══════════════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ ["CORRECTED"]            │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ ["CORRECTED"]            │
│ 2023-01-03 00:00:00 ┆ null        ┆ ["MISSING"]              │
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ ["SUSPECT", "CORRECTED"] │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ []                       │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ ["SUSPECT"]              │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ ["SUSPECT"]              │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ ["SUSPECT"]              │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ ["SUSPECT"]              │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ ["SUSPECT"]              │
└─────────────────────┴─────────────┴──────────────────────────┘

Decoded columns remain registered as flag columns: add_flag(), remove_flag(), and filter_by_flag() all continue to work transparently on the decoded form.

Use encode_flag_column() to round-trip the column back to raw values:

# Round-trip a decoded flag column back to raw integers
tf_encoded = tf_decoded.encode_flag_column("temperature_flags")

shape: (10, 3)
┌─────────────────────┬─────────────┬───────────────────┐
│ time                ┆ temperature ┆ temperature_flags │
│ ---                 ┆ ---         ┆ ---               │
│ datetime[μs]        ┆ f64         ┆ i64               │
╞═════════════════════╪═════════════╪═══════════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5        ┆ 0                 │
│ 2023-01-02 00:00:00 ┆ 21.0        ┆ 0                 │
│ 2023-01-03 00:00:00 ┆ null        ┆ 2                 │
│ 2023-01-04 00:00:00 ┆ 26.0        ┆ 4                 │
│ 2023-01-05 00:00:00 ┆ 24.2        ┆ 0                 │
│ 2023-01-06 00:00:00 ┆ 26.6        ┆ 4                 │
│ 2023-01-07 00:00:00 ┆ 28.4        ┆ 4                 │
│ 2023-01-08 00:00:00 ┆ 30.9        ┆ 4                 │
│ 2023-01-09 00:00:00 ┆ 31.0        ┆ 4                 │
│ 2023-01-10 00:00:00 ┆ 29.1        ┆ 4                 │
└─────────────────────┴─────────────┴───────────────────┘

Integration with `TimeStream` operations¶

The real power of the flagging systems comes into play when integrated with Time-Stream’s operations, such as Quality Control and Infilling. Both accept a flag_params tuple of (flag_column_name, flag_value) that tells the operation to record its outcome directly into an existing flag column, rather than returning a standalone boolean series or leaving the infilled rows unmarked.

This lets you build a single, self-describing TimeFrame where every QC verdict or infilled value is traceable back to the check or method that produced it - no separate bookkeeping required.

Quality control checks write the given flag value onto every row that fails the check:

tf.register_flag_system("QC_FLAGS", ["OUT_OF_RANGE", "SPIKE"])
tf.init_flag_column("QC_FLAGS", "temperature_qc")

# Flag values outside a plausible range
tf.qc_check(
    "range", "temperature", min_value=-30, max_value=50,
    flag_params=("temperature_qc", "OUT_OF_RANGE"),
)

# Flag spikes in the same column
tf.qc_check(
    "spike", "temperature", threshold=10,
    flag_params=("temperature_qc", "SPIKE"),
)

See Quality Control for the full set of available checks and the boolean-series alternative.

Infilling marks rows that were filled in by the chosen method:

tf.register_flag_system("INFILL_FLAGS", ["INFILLED"])
tf.init_flag_column("INFILL_FLAGS", "flow_flags")

tf_infill = tf.infill(
    "linear", "flow", max_gap_size=3,
    flag_params=("flow_flags", "INFILLED"),
)

See Infilling for the full set of infill methods and options.

Because the flag column lives alongside the data, you can then use filter_by_flag() to inspect or exclude affected rows, or decode the column for a human-readable audit trail.

Additional information¶

Bitwise values¶

Bitwise values are a set of integers where each value is a power of 2 (1, 2, 4, 8, 16, etc.). The key feature of bitwise values is that any combination of them results in a unique value. For example, 1 + 4 = 5. The value 5 cannot be produced by any other combination of bitwise numbers. Similarly, the value 13 can only be produced by 1 + 4 + 8.

Bitwise values come from the binary number system, where each bit represents a power of 2.

Decimal	Binary
1	1
2	10
4	100
8	1000
16	10000

With this we can see why combinations of bitwise values are unique. For example, 5 in binary is 101, which corresponds to the combination of 1 (001) and 4 (100).

For Time-Stream, this feature allows any combination of flags from a bitwise flag system to be stored as a single integer value, without losing information about which flags are present. For example, if a data point is flagged as 48 in the CORE_FLAGS flag system, we can deduce it has been both REMOVED (16) and INFILLED (32).

API reference¶

`register_flag_system`(name[, flag_system, ...])	Register a named flag system with the internal flag manager.
`with_flag_system`(name, flag_system[, flag_type])	Return a new TimeFrame, with a flag system registered.
`get_flag_system`(name)	Return a registered flag system.
`init_flag_column`(flag_system_name[, ...])	Add a new column to the TimeFrame DataFrame, setting it as a Flag Column.
`register_flag_column`(column_name, ...)	Mark the specified existing column as a flag column.
`get_flag_column`(flag_column_name)	Look up a registered flag column by name.
`add_flag`(flag_column_name, flag_value[, ...])	Add flag value to flag column, where expression is True.
`remove_flag`(column_name, flag_value[, expr])	Remove a flag value from a flag column, where expression is True.
`filter_by_flag`(flag_column_name, flag[, include])	Return a new TimeFrame filtered to rows that have (or lack) specific flags set.
`decode_flag_column`(flag_column_name)	Decode a flag column from raw values to human-readable flag names.
`encode_flag_column`(flag_column_name)	Encode a decoded flag column back to raw values.

Flagging¶

Why use Time-Stream?¶

Simple example¶

Key benefits¶

In more detail¶

Flag systems¶

bitwise¶

categorical¶

categorical_list¶

Flag columns¶

Adding flags¶

Removing flags¶

Filtering by flag¶

Decoding and encoding flag columns¶

Integration with TimeStream operations¶

Additional information¶

Bitwise values¶

API reference¶

`bitwise`¶

`categorical`¶

`categorical_list`¶

Integration with `TimeStream` operations¶