Skip to contents

Get started with whence

Source: vignettes/articles/get-started.Rmd
get-started.Rmd

A dashboard shows a number. The viewer thinks: “Why is this number smaller than I expected?” whence gives the answer with one click. It shows how many rows each step of the pipeline removed, and the code that makes the number again.

Install

whence is not on CRAN yet. Install it from GitHub:

# install.packages("pak")
pak::pak("tenmeh/whence")

In a Shiny app

Replace textOutput() and renderText() with whenceOutput() and renderWhence(). Give the render a pipeline that ends in one value. You do not need to change the pipeline.

library(shiny)
library(dplyr)
library(whence)

ui <- fluidPage(
  selectInput("cyl", "Cylinders", c(4, 6, 8)),
  whenceOutput("mean_mpg", label = "Mean mpg")
)

server <- function(input, output, session) {
  output$mean_mpg <- renderWhence(
    mtcars |>
      filter(cyl == input$cyl) |>
      filter(hp < 150) |>
      summarise(mpg = round(mean(mpg), 1))
  )
}

shinyApp(ui, server)

The value has a dotted underline. A click on it, or Enter or Space when it has the focus, opens a panel next to it. The panel holds:

  • Where the rows went. The source, then each step with the rows after it and the rows that it removed.
  • Reproduce. The pipeline as code, with each input$name replaced by its current value. The viewer can copy it and run it.
  • View the rows. The rows of the source that make the value. This is the only part that asks the server for something.

The live demo shows all of this in your browser.

Show a value as you want

format in renderWhence() is a function that turns the value into the text to show. A formatter of ‘scales’ works:

output$revenue <- renderWhence(
  sales |> filter(region == input$region) |> summarise(total = sum(revenue)),
  format = scales::label_dollar()
)

So does a function of your own, such as function(x) paste0(round(x), " kg"). It runs on the server, so it can read inputs, for example to choose a currency. With no format, a number gets a comma for each thousand.

With no Shiny

The engine works in the console, which makes it easy to test. whence_trace() takes the same pipeline:

library(dplyr, warn.conflicts = FALSE)
library(whence)

trace <- whence_trace(
  mtcars |>
    filter(cyl == 4) |>
    filter(hp < 90) |>
    summarise(mpg = round(mean(mpg), 1))
)
trace
#> <whence_trace> 29.7
#> 
#> Where the rows went
#>   0 mtcars                  32
#>   1 -> cyl == 4             11  -21 rows (66%)
#>   2 -> hp < 90               5  -6 rows (55%)
#>   3 -> aggregated to 1 row   1
#> 
#> Reproduce
#>   mtcars |>
#>     filter(cyl == 4) |>
#>     filter(hp < 90) |>
#>     summarise(mpg = round(mean(mpg), 1))

One row for each step:

as.data.frame(trace)
#>   step               label  n delta   percent      kind      verb groups shown
#> 1    0              mtcars 32    NA        NA    source      <NA>     NA  TRUE
#> 2    1            cyl == 4 11   -21 -65.62500    filter    filter     NA  TRUE
#> 3    2             hp < 90  5    -6 -54.54545    filter    filter     NA  TRUE
#> 4    3 aggregated to 1 row  1    NA        NA summarise summarise     NA  TRUE

The value, the code, and the rows that make the value:

whence_value(trace)
#> [1] 29.7
cat(whence_code(trace))
#> mtcars |>
#>   filter(cyl == 4) |>
#>   filter(hp < 90) |>
#>   summarise(mpg = round(mean(mpg), 1))
whence_rows(trace)
#>                 mpg cyl  disp hp drat    wt  qsec vs am gear carb
#> Merc 240D      24.4   4 146.7 62 3.69 3.190 20.00  1  0    4    2
#> Fiat 128       32.4   4  78.7 66 4.08 2.200 19.47  1  1    4    1
#> Honda Civic    30.4   4  75.7 52 4.93 1.615 18.52  1  1    4    2
#> Toyota Corolla 33.9   4  71.1 65 4.22 1.835 19.90  1  1    4    1
#> Fiat X1-9      27.3   4  79.0 66 4.08 1.935 18.90  1  1    4    1

Which rows did a step remove?

Each count of removed rows in the waterfall has its percentage, and in the panel it is a button that shows those rows. In the console, give the step to whence_rows(). print() shows the number of each step:

whence_rows(trace, step = 2)
#>                mpg cyl  disp  hp drat    wt  qsec vs am gear carb
#> Datsun 710    22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
#> Merc 230      22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
#> Toyota Corona 21.5   4 120.1  97 3.70 2.465 20.01  1  0    3    1
#> Porsche 914-2 26.0   4 120.3  91 4.43 2.140 16.70  0  1    5    2
#> Lotus Europa  30.4   4  95.1 113 3.77 1.513 16.90  1  1    5    2
#> Volvo 142E    21.4   4 121.0 109 4.11 2.780 18.60  1  1    4    2

Tables

A pipeline that ends in a summary gives a table. whenceTableOutput() and renderWhenceTable() show it, and the first cell of each row opens the panel of that row: the rows of its group at each step, the code of the row, and its rows.

ui <- fluidPage(whenceTableOutput("by_cyl", label = "Cars by cylinders"))

server <- function(input, output, session) {
  output$by_cyl <- renderWhenceTable(
    mtcars |>
      filter(hp > 90) |>
      group_by(cyl) |>
      summarise(cars = n(), mpg = mean(mpg)),
    format = list(mpg = function(x) format(round(x, 1)))
  )
}

In the console, whence_table() traces the table, and whence_group() explains one row:

by_cyl <- whence_table(
  mtcars |>
    filter(hp > 90) |>
    group_by(cyl) |>
    summarise(cars = n(), mpg = mean(mpg)) |>
    mutate(share = cars / sum(cars))
)
six <- whence_group(by_cyl, cyl = 6)
six
#> <whence_group> cyl = 6
#> 
#> Where the rows went
#>   0 mtcars                             7
#>   1 -> hp > 90                         7
#>   2 -> grouped by cyl                  7
#>   3 -> summarised into this row        1
#>   4 -> mutate(share = cars/sum(cars))  1
#> 
#> Reproduce
#>   mtcars |>
#>     filter(hp > 90) |>
#>     group_by(cyl) |>
#>     summarise(cars = n(), mpg = mean(mpg)) |>
#>     mutate(share = cars/sum(cars)) |>
#>     filter(cyl == 6)
whence_rows(six, step = 1)
#>  [1] mpg  cyl  disp hp   drat wt   qsec vs   am   gear carb
#> <0 rows> (or 0-length row.names)

A row can be found by its number, as in whence_group(by_cyl, 2), or by its group values, as above.

The code of the row keeps share right: it is the code of the table, and a last filter() that keeps the row.

Plots

whencePlotOutput() and renderWhencePlot() draw a ‘ggplot2’ plot of a summary. A click on a bar or a point opens the panel of its row. The x aesthetic can be computed from the columns of the summary, as in aes(reorder(cyl, mpg), mpg). Under the plot, “Show the data” gives the table, so a keyboard user can open each panel too.

output$by_cyl_plot <- renderWhencePlot(
  mtcars |>
    filter(hp > 90) |>
    group_by(cyl = factor(cyl)) |>
    summarise(mpg = mean(mpg)),
  plot = function(data) ggplot(data, aes(cyl, mpg)) + geom_col()
)

Follow the viewer

whence_opened() gives a reactive with the panel that the viewer has open. While a row of a table is open, it gives the number of the row and the row itself, so the app can show more about it:

server <- function(input, output, session) {
  output$by_cyl <- renderWhenceTable(
    mtcars |> group_by(cyl) |> summarise(cars = n())
  )
  opened <- whence_opened("by_cyl")
  output$detail <- renderText({
    req(opened())
    paste("Looking at the cars with", opened()$data$cyl, "cylinders.")
  })
}

Summaries are not attrition

When summarise() turns 1,204 rows into one, the panel says “aggregated to 1 row”. It does not say “-1,203 rows”, because no rows were lost. After a group_by() it says “grouped into 14 rows”. A group_by() is shown only when it changes the groups.

whence_trace(
  mtcars |>
    group_by(cyl) |>
    summarise(mpg = mean(mpg)) |>
    filter(cyl == 6) |>
    pull(mpg)
)
#> <whence_trace> 19.74286
#> 
#> Where the rows went
#>   0 mtcars                  32
#>   1 -> grouped by cyl       32
#>   2 -> grouped into 3 rows   3
#>   3 -> cyl == 6              1  -2 rows (67%)
#> 
#> Reproduce
#>   mtcars |>
#>     group_by(cyl) |>
#>     summarise(mpg = mean(mpg)) |>
#>     filter(cyl == 6) |>
#>     pull(mpg)

When a pipeline is not traced

whence knows the verbs of ‘dplyr’ and two pivots of ‘tidyr’ (the list is on the help page of whence_trace()). Any other call ends the chain, and the trace says where:

cylinders <- data.frame(cyl = c(4, 6, 8), engine = c("small", "mid", "big"))
whence_trace(
  mtcars |>
    filter(hp > 100) |>
    inner_join(cylinders, by = "cyl") |>
    count()
)
#> <whence_trace> 23
#> 
#> Where the rows went
#>   0 inner_join(filter(mtcars, hp > 100), cylinders, by = "cyl")  23
#>   1 -> aggregated to 1 row                                        1
#>   Steps before inner_join() are not shown, because whence does not know inner_join().
#> 
#> Reproduce
#>   inner_join(filter(mtcars, hp > 100), cylinders, by = "cyl") |>
#>     count()

When the rows cannot be found

The waterfall needs only counts. To find which rows make the value, whence_rows() runs the steps again with a hidden id column. Some steps break the link between a row and the source. After distinct(), a pivot of ‘tidyr’, or a step that is not a known verb, whence_rows() gives NULL with a warning that names the step. The waterfall still works.

trace <- whence_trace(
  mtcars |>
    filter(hp > 100) |>
    distinct(cyl) |>
    count()
)
rows <- whence_rows(trace)
#> Warning: Row attribution unavailable past distinct().

Limits

  • Database-backed pipelines are not supported. With ‘dbplyr’ the steps run as SQL only at collect(), so there are no counts for each step. whence stops with an error, and shows no waterfall that would mislead.
  • Joins are not traced. A join ends the chain: the join and everything upstream of it become the source.
  • Window functions are counted correctly, but their row attribution has no clear meaning.
  • The verbs of ‘data.table’, ‘arrow’ and ‘duckplyr’ are not known.
  • A table or a plot is explained row by row when it is the result of a summary step that keeps its group columns. A plot must be a ‘ggplot2’ plot of that table.