---
title: "Tracking how a literature changes between retrievals"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Tracking how a literature changes between retrievals}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = FALSE, comment = "")
# Console colour carries no meaning on a rendered page. pkgdown turns it on for
# its own build, and the escape sequences then reach the reader as literal text,
# so colour is switched off here for a plain vignette render and a site build
# alike. The fixed width keeps tibbles inside the documentation column.
options(cli.num_colors = 1, cli.hyperlink = FALSE, crayon.enabled = FALSE,
        width = 80)
# Print data frames and tibbles as formatted tables.
local({
  kp <- function(x, ...) {
    if (any(vapply(x, is.list, logical(1)))) return(knitr::normal_print(x))
    knitr::knit_print(knitr::kable(x))
  }
  for (cls in c("data.frame", "tbl_df", "tbl")) {
    registerS3method("knit_print", cls, kp, envir = asNamespace("knitr"))
  }
})
```

```{r setup}
library(scopusflow)
```

A literature is a moving target. Run the same search a few months apart and the
result will have grown, and perhaps lost a record that was re-indexed. This
article shows how to see exactly what changed and how to merge retrievals safely.
It runs offline on the bundled `example_records`, a corpus of 138 real journal
articles the package ships because 'Scopus' records may not be redistributed.
That corpus is a complete harvest of one query from 2015 to 2024, so a pull that
stopped at 2023 and a later one that reaches 2024 are both genuine slices of the
same search.

## The baseline

The first retrieval ran at the end of 2023 and returned everything published up
to then.

```{r}
baseline <- example_records[example_records$year <= 2023, ]
nrow(baseline)
```

## A later retrieval

A year on, the search is repeated. It now picks up the 2024 papers, and one
record that was present the first time has since been re-indexed and no longer
matches.

```{r}
later <- example_records[-1, ]
nrow(later)
```

## What changed

`scopus_diff_dois()` reports which DOIs were added, removed or unchanged between
the two retrievals, and prints the counts in each category.

```{r}
changes <- scopus_diff_dois(old = baseline, new = later)
print(changes)
```

The newly indexed papers come back as `added`, the records present both times as
`unchanged`, and anything dropped from the later pull as `removed`. The counts
work out at fourteen added, one removed and 112 unchanged. Fourteen are added
because that is how many of the 2024 papers carry a DOI, and the re-indexed
record takes the unchanged count from 113 down to 112. Records without a DOI
cannot be tracked this way at all, which is one reason to prefer the 'Scopus'
identifier when there is one.

To act on one category, filter the table, which is an ordinary tibble.

```{r}
head(changes[changes$status == "added", ])
```

## Merging without duplicates

To keep a cumulative set across retrievals, combine them. `scopus_combine()`
renumbers the records and, with `dedupe = TRUE`, keeps each one once by 'Scopus'
identifier or DOI, so the records the two pulls share are not doubled.

```{r}
combined <- scopus_combine(baseline, later, dedupe = TRUE)
nrow(combined)
```

That is 149 rows for 138 distinct articles, and the gap is instructive. These
records carry no 'Scopus' identifier, never having come from 'Scopus', so
de-duplication falls back to the DOI. The eleven that arrived without one have
no key to match on, and so survive in both copies. A live harvest carries an
identifier on every record, so the same call on two real pulls returns each
article once.

The base `c()` method concatenates record sets directly, renumbering but without
de-duplicating, so it is the building block that `scopus_combine()` adds the
duplicate handling to.

```{r}
stacked <- c(baseline, later)
nrow(stacked)
```

## Keeping a record of each pull

Saving each retrieval lets you compare against it next time. The `.rds` form
round-trips exactly.

```{r}
path <- file.path(tempdir(), "baseline.rds")
write_scopus_records(baseline, path)
identical(read_scopus_records(path), baseline)
```

A live retrieval also carries the date it was taken, as the `retrieved_at`
attribute, together with the `scopusflow` version that took it. That matters
for this workflow in particular, because `citations` is a snapshot value that
keeps moving, so a difference between two pulls only means something once you
know how far apart they were. Both survive the `.rds` form and neither survives
`.csv`, which is a table of columns. The bundled corpus used here carries
neither, which is why it round-trips identically above.

In a live setting the later retrieval would come from the API, where here it is
a slice of the bundled corpus, and everything else would be as above. Both pulls
could then say when they were taken.

```{r eval = FALSE}
later <- scopus_fetch("graphene supercapacitor", field = "TITLE-ABS-KEY")
attr(later, "retrieved_at")
scopus_diff_dois(old = read_scopus_records(path), new = later)
```