maestro 1.4.0 lets R pipeline authors log row counts and API calls for each run
maestro 1.4.0 lets authors of R pipelines record custom numbers, such as rows read or API calls, with a new update_metric() function. The values stay out of each step's return value, so downstream steps never receive them.
The Scientist · Science desk
In the release post's example, the get_metrics() table had 3 rows after the first run_schedule() call and 6 rows after the second. Metrics from both runs were kept and told apart by run_id.
| Measure | Value | As of | Claim |
|---|---|---|---|
| After first run_schedule() | 3 metric rows | 12 | |
| After second run_schedule() | 6 metric rows | 13 |
What happened
- Before this release maestro tracked execution time and success or failure, with no good way to answer how many rows a pipeline processed or how many API calls it took.
- After a scheduled run, get_metrics() returns a long data.frame with one row per metric, giving the pipeline name, the run ID, the metric name and its numeric value.
- Passing increment = TRUE adds to a metric's current value instead of overwriting it, so a loop can count API calls or retry attempts, and a negative value decrements.
- Outside run_schedule(), such as when a developer tests a pipeline function interactively, update_metric() prints an informational message and records nothing.
Why it matters
- capability Operators can compare a successful run against the rows it processed. Status and timing data alone could not show a run that succeeded on far less data than usual.
- decision Each step in a DAG logs under its own run ID, so matching an extract count to a load count from one pass means choosing a key other than run_id.
- constraint A metric holds one number per name per run, so a latency distribution or a list of values has to be reduced to a single figure before it can be logged.
The release post's example is small. The extract step builds a five-row data frame inside the function as a stand-in for an API call [10] and records rows_read as 5 and bytes_read as 1648 [9]. The second figure comes from as.numeric(object.size(df)) [10]. In our view that measures the object R holds. Against a real API, the bytes transferred would be a different number.
Within one execution, a second call with the same name overwrites the first: the last write wins [11]. Across executions nothing is overwritten. The second run_schedule() call in the post took the output from 3 rows [12] to 6 [13]. That is 3 new rows per call for this two-step DAG [17].
The post calls the plain data.frame a natural fit for monitoring pipeline health over time [14]. The printed table has four columns, and none is a timestamp [5]. Execution time and success or failure sit in get_status() and get_schedule() [15]; nothing in the examples shows whether run_id also appears there. A trend line needs that link, or a timestamp the author adds when writing the table to a database or file.
We think the side channel is sound for a DAG. A step's return value stays the contract between steps [4], and the counters are set from inside the step with just a name and a value [1]. The condition is careful naming, since the metric name is the only label a row carries beyond pipeline and run [5]. maestro is a pipeline orchestration package, and 1.4.0 installs from CRAN with install.packages("maestro") [16].
What to watch
- Whether a later release adds a timestamp column to get_metrics() or a shared key tying together the steps of one DAG run.
- Package documentation on whether recorded metrics persist beyond the R session that ran the schedule.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap−5
- Incentives
- Insufficient
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
The maestro 1.4.0 release lets users record custom metrics from pipelines by calling update_metric() from inside a pipeline with the name of the metric and its value.
- [2]
Before 1.4.0, maestro tracked execution time and success/failure, but there was no good way to answer how many rows a pipeline processed or how many API calls it took.
- [3]
A metric is any value that can be represented as a single number, such as rows processed, bytes read, API calls made, or attempts on a task.
- [4]
Metrics are recorded on the side. They are not part of a pipeline's return value, so they never get passed downstream in a DAG.
- [5]
After running the schedule, get_metrics() returns a long data.frame with the columns pipe_name, run_id, metric and value; the printed examples show no timestamp column.
- [6]
Setting increment = TRUE adds value to the metric's current value instead of overwriting it; if the metric has not been set in the current run the first call sets it, and a negative value decrements. The post's examples are counting API calls in a loop and attempts on a flaky task.
- [7]
Metrics are only recorded when a pipeline is executed by run_schedule(); calling update_metric() while developing or testing a pipeline function interactively emits an informational message and does nothing.
- [8]
Each pipeline gets its own run_id, even within a DAG; in the post's first run, extract_flights had run_id vPnJoC and load_flights had run_id ulLl5m.
- [9]
In the post's example output, extract_flights recorded rows_read as 5 and bytes_read as 1648.
- [10]
In the example, extract_flights builds a five-row data frame inside the function as a stand-in for fetching data from an API, and computes bytes_read as as.numeric(object.size(df)).
- [11]
Metrics belong to a single execution: each run gets a new run_id and its metrics are stored separately from earlier runs. Within one execution, calling update_metric() again with the same name overwrites the earlier value (last write wins).
- [12]
After the first run_schedule() call, get_metrics() returned a table of 3 rows.
- [13]
After a second run_schedule() call, get_metrics() returned 6 rows: the metrics from both runs were retained and distinguished by run_id.
- [14]
Because get_metrics() returns a plain data.frame, it is easy to summarise further or write out to a database or file, which the post says makes it a natural fit for monitoring pipeline health over time.
- [15]
Execution time and success/failure are already available via get_status() and get_schedule(), so there is no need to record those with update_metric().
- [16]
maestro is a pipeline orchestration package, and version 1.4.0 can be installed from CRAN with install.packages("maestro").
- [17]
The second run_schedule() call added 3 rows to the get_metrics() output for the two-step example DAG.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- r-bloggers.commaestro 1.4.0 adds custom pipeline metrics
1 article · October 14, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Entities
- maestroFollow
- CRANFollow
- R-bloggersFollow