Quick start#
Create and work with timeseries data using the Time-Stream package.
import time_stream as ts
Create a TimeFrame#
Create sample data in a Polars DataFrame:
dates = [datetime(2023, 1, 1) + timedelta(days=i) for i in range(10)]
df = pl.DataFrame(
{
"time": dates,
"temperature": [20.5, 21.0, 19.1, 26.0, 24.2, 26.6, 28.4, 30.9, 31.0, 29.1],
"precipitation": [0.0, 0.0, 5.1, 10.2, 2.0, 0.2, 0.0, 3.0, 1.6, 0.0],
}
)
Now wrap the Polars DataFrame in a TimeFrame, which adds specialized functionality
for time series operations:
tf = ts.TimeFrame(df=df, time_name="time")
print(tf)
<time_stream.TimeFrame> Size (estimated): 288.00 B
Time properties:
Time column : time [2023-01-01 00:00:00, ..., 2023-01-10 00:00:00]
Type : Datetime(time_unit='us', time_zone=None)
Resolution : PT0.000001S
Offset : None
Alignment : PT0.000001S
Periodicity : PT0.000001S
Anchor : start
Columns:
temperature : Float64 80.00 B [20.5, ..., 29.1]
precipitation : Float64 80.00 B [0.0, ..., 0.0]
DataFrame:
shape: (10, 3)
┌─────────────────────┬─────────────┬───────────────┐
│ time ┆ temperature ┆ precipitation │
│ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ f64 ┆ f64 │
╞═════════════════════╪═════════════╪═══════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5 ┆ 0.0 │
│ 2023-01-02 00:00:00 ┆ 21.0 ┆ 0.0 │
│ 2023-01-03 00:00:00 ┆ 19.1 ┆ 5.1 │
│ 2023-01-04 00:00:00 ┆ 26.0 ┆ 10.2 │
│ 2023-01-05 00:00:00 ┆ 24.2 ┆ 2.0 │
│ 2023-01-06 00:00:00 ┆ 26.6 ┆ 0.2 │
│ 2023-01-07 00:00:00 ┆ 28.4 ┆ 0.0 │
│ 2023-01-08 00:00:00 ┆ 30.9 ┆ 3.0 │
│ 2023-01-09 00:00:00 ┆ 31.0 ┆ 1.6 │
│ 2023-01-10 00:00:00 ┆ 29.1 ┆ 0.0 │
└─────────────────────┴─────────────┴───────────────┘
Note
The time column must be a Polars Date or Datetime column, either without a time zone or in UTC. Time
zones with daylight saving aren’t supported, as their days aren’t all 24 hours long. Convert such a column to UTC
with pl.col("time").dt.convert_time_zone("UTC"), or keep the local clock times and remove the time zone with
pl.col("time").dt.replace_time_zone(None).
With Time Properties#
The TimeFrame object can configure important properties about the time aspect of your data.
More information about these properties and concepts can be found on the concepts page.
Here, we will show some basic usage of these time properties.
Resolution, Offset, Periodicity and Time Anchor#
Without specifying resolution or periodicity, the default initialisation sets these properties to
1 microsecond, to account for any set of datetime values.
The default is for no offset. The time_anchor is set to start:
print("resolution:", tf.resolution)
print("offset:", tf.offset)
print("periodicity:", tf.periodicity)
print("time anchor:", tf.time_anchor)
resolution: PT0.000001S
offset: None
periodicity: PT0.000001S
time anchor: start
Although the default of 1 microsecond will account for any datetime values, for more control over certain
time series functionality it is important to specify the actual resolution, offset and periodicity if known.
These properties can be provided as an ISO 8601 duration string, e.g. P1D (1 day) or PT15M (15 minutes).
The time_anchor property can be set to start, end, or point.
Again, more detail can be found on the concepts page about all these properties.
Resolution#
For most cases, it is sufficient to just specify the resolution. The offset will default to “no offset”, and the
periodicity will be set to the same as the resolution.
tf = ts.TimeFrame(df=df, time_name="time", resolution="P1D")
print("resolution:", tf.resolution)
print("offset:", tf.offset)
print("periodicity:", tf.periodicity)
resolution: P1D
offset: None
periodicity: P1D
Offset#
The next most common modification might be to specify an offset. This is where your data is measured at a
point in time offset from the “natural boundary” of the resolution (more info here:
concepts page). The periodicity is automatically built from the resolution + offset, to
specify that we only expect 1 value within those points in time.
tf = ts.TimeFrame(df=df, time_name="time", resolution="P1D", offset="+T9H")
print("resolution:", tf.resolution)
print("offset:", tf.offset)
print("periodicity:", tf.periodicity)
resolution: P1D
offset: +T9H
periodicity: P1D+T9H
Periodicity#
Finally, you may have data that is measured at a given resolution, but you only expect 1 value in a different period of
time. This is when you would specify a specific periodicity. The classic hydrological example would be where you
have an annual-maximum (AMAX) timeseries, where the measured data is a daily resolution, but we only expect 1 value
per year.
tf = ts.TimeFrame(
df=df,
time_name="time",
resolution="P1D", # daily sampling interval
offset="+T9H", # readings taken at 09:00
periodicity="P1Y+9MT9H", # at most one value per water year (1 Oct 09:00)
)
print("resolution:", tf.resolution)
print("offset:", tf.offset)
print("periodicity:", tf.periodicity)
resolution: P1D
offset: +T9H
periodicity: P1Y+9MT9H
Duplicate Detection#
TimeFrame automatically checks for rows with duplicates in the specified time column.
You have control over what the model should do when it detects rows with duplicate time values.
Consider this DataFrame with duplicate time values:
shape: (10, 3)
┌─────────────────────┬─────────────┬───────────────┐
│ time ┆ temperature ┆ precipitation │
│ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ i64 ┆ i64 │
╞═════════════════════╪═════════════╪═══════════════╡
│ 2023-01-01 00:00:00 ┆ 20 ┆ null │
│ 2023-01-01 00:00:00 ┆ null ┆ 0 │
│ 2023-02-01 00:00:00 ┆ 19 ┆ 5 │
│ 2023-03-01 00:00:00 ┆ 26 ┆ 10 │
│ 2023-04-01 00:00:00 ┆ 24 ┆ 2 │
│ 2023-05-01 00:00:00 ┆ 26 ┆ 0 │
│ 2023-06-01 00:00:00 ┆ 28 ┆ null │
│ 2023-06-01 00:00:00 ┆ 30 ┆ 3 │
│ 2023-06-01 00:00:00 ┆ null ┆ 4 │
│ 2023-07-01 00:00:00 ┆ 29 ┆ 0 │
└─────────────────────┴─────────────┴───────────────┘
The following strategies are available to use with the on_duplicate argument:
Error (Default):
on_duplicate="error"
Raises an error when duplicate rows are found. This is the default behavior to ensure data integrity.
try:
ts.TimeFrame(df, "time", on_duplicates="error")
except DuplicateTimeError as err:
print(err)
Duplicate time values found. A TimeFrame must have unique time values. Options for dealing with duplicate rows include: ['drop', 'keep_first', 'keep_last', 'error', 'merge'].
Keep First:
on_duplicate="keep_first"
For a given group of rows with the same time value, keeps only the first row and discards the others.
tf = ts.TimeFrame(df, "time", on_duplicates="keep_first")
print(tf.df)
shape: (7, 3)
┌─────────────────────┬─────────────┬───────────────┐
│ time ┆ temperature ┆ precipitation │
│ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ i64 ┆ i64 │
╞═════════════════════╪═════════════╪═══════════════╡
│ 2023-01-01 00:00:00 ┆ 20 ┆ null │
│ 2023-02-01 00:00:00 ┆ 19 ┆ 5 │
│ 2023-03-01 00:00:00 ┆ 26 ┆ 10 │
│ 2023-04-01 00:00:00 ┆ 24 ┆ 2 │
│ 2023-05-01 00:00:00 ┆ 26 ┆ 0 │
│ 2023-06-01 00:00:00 ┆ 28 ┆ null │
│ 2023-07-01 00:00:00 ┆ 29 ┆ 0 │
└─────────────────────┴─────────────┴───────────────┘
Keep Last:
on_duplicate="keep_last"
For a given group of rows with the same time value, keeps only the last row and discards the others.
tf = ts.TimeFrame(df, "time", on_duplicates="keep_last")
print(tf.df)
shape: (7, 3)
┌─────────────────────┬─────────────┬───────────────┐
│ time ┆ temperature ┆ precipitation │
│ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ i64 ┆ i64 │
╞═════════════════════╪═════════════╪═══════════════╡
│ 2023-01-01 00:00:00 ┆ null ┆ 0 │
│ 2023-02-01 00:00:00 ┆ 19 ┆ 5 │
│ 2023-03-01 00:00:00 ┆ 26 ┆ 10 │
│ 2023-04-01 00:00:00 ┆ 24 ┆ 2 │
│ 2023-05-01 00:00:00 ┆ 26 ┆ 0 │
│ 2023-06-01 00:00:00 ┆ null ┆ 4 │
│ 2023-07-01 00:00:00 ┆ 29 ┆ 0 │
└─────────────────────┴─────────────┴───────────────┘
Drop:
on_duplicate="drop"
Removes all rows that have duplicate timestamps. This strategy is appropriate when you are unsure of the integrity of duplicate rows and only want unique, unambiguous data.
tf = ts.TimeFrame(df, "time", on_duplicates="drop")
print(tf.df)
shape: (5, 3)
┌─────────────────────┬─────────────┬───────────────┐
│ time ┆ temperature ┆ precipitation │
│ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ i64 ┆ i64 │
╞═════════════════════╪═════════════╪═══════════════╡
│ 2023-02-01 00:00:00 ┆ 19 ┆ 5 │
│ 2023-03-01 00:00:00 ┆ 26 ┆ 10 │
│ 2023-04-01 00:00:00 ┆ 24 ┆ 2 │
│ 2023-05-01 00:00:00 ┆ 26 ┆ 0 │
│ 2023-07-01 00:00:00 ┆ 29 ┆ 0 │
└─────────────────────┴─────────────┴───────────────┘
Merge:
on_duplicate="merge"
For a given group of rows with the same time value, performs a merge of all rows. This combines values with a top-down approach that preserves the first non-null value for each column.
tf = ts.TimeFrame(df, "time", on_duplicates="merge")
print(tf.df)
shape: (7, 3)
┌─────────────────────┬─────────────┬───────────────┐
│ time ┆ temperature ┆ precipitation │
│ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ i64 ┆ i64 │
╞═════════════════════╪═════════════╪═══════════════╡
│ 2023-01-01 00:00:00 ┆ 20 ┆ 0 │
│ 2023-02-01 00:00:00 ┆ 19 ┆ 5 │
│ 2023-03-01 00:00:00 ┆ 26 ┆ 10 │
│ 2023-04-01 00:00:00 ┆ 24 ┆ 2 │
│ 2023-05-01 00:00:00 ┆ 26 ┆ 0 │
│ 2023-06-01 00:00:00 ┆ 28 ┆ 3 │
│ 2023-07-01 00:00:00 ┆ 29 ┆ 0 │
└─────────────────────┴─────────────┴───────────────┘
Misaligned Row Detection#
As part of the resolution checks, misaligned rows (rows with a different resolution to that expected) are detected
automatically. By default an error is raised, however if on_misaligned_rows="resolve" is set on creation of the
TimeFrame, then any misaligned rows will be automatically removed.
Note
A resolution must be provided to the TimeFrame initialisation for the misaligned row detection
to work. If not provided, a default resolution of 1 microsecond is assigned, which will pass any resolution checks.
The following strategies are available to use with the on_misaligned_rows argument:
Error (Default):
on_misaligned_rows="error"
Raises an error when misaligned rows are found within the time series.
tf = ts.TimeFrame(df, "time", "PT30M", on_misaligned_rows="error")
The following error will be shown if misaligned rows are found. Note that this example assumes PT30M data:
Time values are not aligned to resolution[+offset]: PT30M
Resolve:
on_misaligned_rows="resolve"
Any rows identified as being misaligned are removed and the TimeFrame data is updated.
tf = ts.TimeFrame(df, "time", "PT30M", on_misaligned_rows="resolve")
initial data:
shape: (9, 1)
┌─────────────────────┐
│ time │
│ --- │
│ datetime[μs] │
╞═════════════════════╡
│ 2020-01-01 00:00:00 │
│ 2020-01-01 00:30:00 │
│ 2020-01-01 01:00:00 │
│ 2020-01-01 01:30:00 │
│ 2020-01-01 01:31:00 │
│ 2020-01-01 01:32:00 │
│ 2020-01-01 01:33:00 │
│ 2020-01-01 02:00:00 │
│ 2020-01-01 02:30:00 │
└─────────────────────┘
resolved data, misaligned rows removed:
shape: (6, 1)
┌─────────────────────┐
│ time │
│ --- │
│ datetime[μs] │
╞═════════════════════╡
│ 2020-01-01 00:00:00 │
│ 2020-01-01 00:30:00 │
│ 2020-01-01 01:00:00 │
│ 2020-01-01 01:30:00 │
│ 2020-01-01 02:00:00 │
│ 2020-01-01 02:30:00 │
└─────────────────────┘
With Metadata#
The TimeFrame object can hold metadata to describe your data.
This can be metadata about the time series dataset as a whole, or about the individual columns. Keeping the metadata
and the data together in one object like this can help simplify downstream processes,
such as derivation functions, running infilling routines, plotting data, etc.
Dataset-level metadata can be set with the with_metadata() method:
metadata = {"location": "UKCEH Wallingford", "station_id": "ABC123"}
tf = tf.with_metadata(metadata)
print(tf.metadata)
Column-level metadata can be set with the with_column_metadata() method:
column_metadata = {
"temperature": {"units": "°C", "description": "Average temperature"},
"precipitation": {"units": "mm", "description": "Precipitation amount", "instrument_type": "Tipping bucket"},
}
tf = tf.with_column_metadata(column_metadata)
print(tf.column_metadata)
Metadata can be accessed via the metadata (dataset-level)
and column_metadata (column-level) attributes:
print("all dataset metadata:", tf.metadata)
print("one key:", tf.metadata["location"])
print("all column metadata:", tf.column_metadata)
print("one column:", tf.column_metadata["temperature"])
print("one column key:", tf.column_metadata["temperature"]["units"])
all dataset metadata: {'location': 'UKCEH Wallingford', 'station_id': 'ABC123'}
one key: UKCEH Wallingford
all column metadata: {'time': {}, 'temperature': {'units': '°C', 'description': 'Average temperature'}, 'precipitation': {}}
one column: {'units': '°C', 'description': 'Average temperature'}
one column key: °C
Data Access and Update#
Data Selection#
The underlying Polars DataFrame is accessed via the df property
tf.df
You can create new TimeFrame objects as a selection, using the
select() method, or via indexing syntax:
df = tf.df # the whole thing, as a Polars DataFrame
selected = tf.select(["temperature"]) # a column subset, still a TimeFrame
selected = tf[["temperature"]] # the same, via indexing
print(type(selected))
print(selected.df)
<class 'time_stream.base.TimeFrame'>
shape: (10, 2)
┌─────────────────────┬─────────────┐
│ time ┆ temperature │
│ --- ┆ --- │
│ datetime[μs] ┆ f64 │
╞═════════════════════╪═════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5 │
│ 2023-01-02 00:00:00 ┆ 21.0 │
│ 2023-01-03 00:00:00 ┆ 19.1 │
│ 2023-01-04 00:00:00 ┆ 26.0 │
│ 2023-01-05 00:00:00 ┆ 24.2 │
│ 2023-01-06 00:00:00 ┆ 26.6 │
│ 2023-01-07 00:00:00 ┆ 28.4 │
│ 2023-01-08 00:00:00 ┆ 30.9 │
│ 2023-01-09 00:00:00 ┆ 31.0 │
│ 2023-01-10 00:00:00 ┆ 29.1 │
└─────────────────────┴─────────────┘
Note
The primary time column is automatically maintained in any selection.
Data Update#
If you need to make changes to the underlying Polars DataFrame, use the with_df() method.
This performs some checks on the new DataFrame to check the integrity of the time data has been maintained, and
returns a new TimeFrame object with the updated data.
new_df = tf.df.with_columns((pl.col("temperature") * 1.8 + 32).alias("temperature_f"))
tf = tf.with_df(new_df)
print(tf.df)
shape: (10, 4)
┌─────────────────────┬─────────────┬───────────────┬───────────────┐
│ time ┆ temperature ┆ precipitation ┆ temperature_f │
│ --- ┆ --- ┆ --- ┆ --- │
│ datetime[μs] ┆ f64 ┆ f64 ┆ f64 │
╞═════════════════════╪═════════════╪═══════════════╪═══════════════╡
│ 2023-01-01 00:00:00 ┆ 20.5 ┆ 0.0 ┆ 68.9 │
│ 2023-01-02 00:00:00 ┆ 21.0 ┆ 0.0 ┆ 69.8 │
│ 2023-01-03 00:00:00 ┆ 19.1 ┆ 5.1 ┆ 66.38 │
│ 2023-01-04 00:00:00 ┆ 26.0 ┆ 10.2 ┆ 78.8 │
│ 2023-01-05 00:00:00 ┆ 24.2 ┆ 2.0 ┆ 75.56 │
│ 2023-01-06 00:00:00 ┆ 26.6 ┆ 0.2 ┆ 79.88 │
│ 2023-01-07 00:00:00 ┆ 28.4 ┆ 0.0 ┆ 83.12 │
│ 2023-01-08 00:00:00 ┆ 30.9 ┆ 3.0 ┆ 87.62 │
│ 2023-01-09 00:00:00 ┆ 31.0 ┆ 1.6 ┆ 87.8 │
│ 2023-01-10 00:00:00 ┆ 29.1 ┆ 0.0 ┆ 84.38 │
└─────────────────────┴─────────────┴───────────────┴───────────────┘