Decision models like Jev have proven to be terrifically useful as an “LLM Judge” for evaluating AI systems. The characteristics of a generalized classifier make it well-suited for the arbitrary multi-choice decisions that are needed for a strong AI eval set. For search evaluations, however, the ground truth we’re after is not typically a classification, but instead a ranked set of top search results — which doesn’t directly fit into the decision-model paradigm.
Many Search teams use an LLM to process a limited set of candidate results and rerank them into an ordered sequence of ground-truth search results. Blitzrank is a “tournament graph framework” for estimating those ground-truth rankings as efficiently as possible. In this post I’ll show how we can replace Blitzrank’s RankGPT “oracle” with pairwise Jev comparisons to make Blitzrank 15x faster without sacrificing relevance.
Results
I forked Contextual AI’s Blitzrank repo, and compared the relevance of the reranked results produced by gpt-6-luna with jev-1.13.0 across 8 open-source datasets. On average, Jev returned its rankings 15x faster than Luna; the rankings were marginally more relevant, and slightly cheaper.
The original paper calculates relevance across 13 open-source datasets… I made it through 8 of them before burning through all of my openai api credits, which is why the others are omitted.
| Metric | Jev | GPT-6 Luna |
|---|---|---|
| nDCG@10 | 0.5477 | 0.5357 |
| Cost/query | $0.0058 | $0.0082 |
| Latency per API call | 0.60s | 8.74s |
| Time/query (all calls) | 3.7s | 55.5s |
Results by dataset
Time/query
nDCG@10
Cost/query
Luna is slightly cheaper on datasets where the documents are short — dl19, signal1m and dbpedia; the longer-document corpora (trec-news, robust04, touche2020) run ~40–70% more expensive.
How we replace RankGPT with Jev
Let’s compare how Jev differs from RankGPT by stepping through a single query from the BeIR/dbpedia-entity dataset. The Blitzrank paper (and my comparison) begins by grabbing the top candidates by bm25 scores. Here’s each of the top-20 documents for the query, "Members of the band Jefferson Airplane."
Top bm25 candidates
| # | Doc id | BM25 Score | Document |
|---|---|---|---|
| 1 | Jefferson_Airplane_Loves_You | 14.23 |
Title: Jefferson Airplane Loves You Content: Jefferson Airplane Loves You is a three-CD boxed set of recordings by the San Francisco rock band Jefferson Airplane with extensive liner notes by Jeff Tamarkin, author of the Jefferson Airplane history Got a Revolution: The Turbulent Flight of Jefferson Airplane.Many of the tracks are previously unreleased live recordings or studio rehearsals, but several are lifted from prior Jefferson Airplane albums. |
| 2 | Through_the_Looking_Glass | 14.04 |
Title: Through the Looking Glass (Jefferson Airplane album) Content: Through the Looking Glass is an Italian release of live performances by the San Francisco rock band Jefferson Airplane.The collection was originally issued as a two LP (long-playing vinyl) set on the Get Back label. The CD was released in 2002 on the Metrodome (DTK) label.Three of the tracks are credited to the “Steel Riders”, a trio consisting of Jefferson Airplane band members Marty Balin, Jorma Kaukonen and Skip Spence. |
| 3 | Jefferson_Starship | 13.91 |
Title: Jefferson Starship Content: Jefferson Starship is an American rock band formed in the early 1970s by several members of the former psychedelic rock group Jefferson Airplane. The band has undergone several major changes in personnel and genres through the years while retaining the same Jefferson Starship name. |
| 4 | The_Worst_of_Jefferson_Airplane | 13.86 |
Title: The Worst of Jefferson Airplane Content: The Worst of Jefferson Airplane is the first compilation album from the rock band Jefferson Airplane, released in November 1970 as RCA Victor LSP-4459. The album features all of Jefferson Airplane’s hit singles up to that point. It peaked at #12 on the Billboard 200 in 1971 and has since gone platinum. |
| 5 | Somebody_to_Love_(Jefferson_Airplane_song) | 13.56 |
Title: Somebody to Love (Jefferson Airplane song) Content: “Somebody to Love”, originally titled “Someone to Love”, is a rock song that was written by Darby Slick, and originally recorded by 1960s folk rock band, The Great Society, and later by the psychedelic counterculture rock band Jefferson Airplane. Rolling Stone magazine ranked Jefferson Airplane’s version No. 274 on their list of the 500 Greatest Songs of All Time. |
| 6 | The_Essential_Jefferson_Airplane | 13.42 |
Title: The Essential Jefferson Airplane Content: The Essential Jefferson Airplane is a compilation of music from the San Francisco rock band Jefferson Airplane spanning its entire career, excluding the brief reunion in 1989 (despite that album now being under common ownership with the rest of the band’s catalog).It follows their development, in more-or-less chronological order, from their beginnings in folk-rock, through psychedelia, to conventional rock genres. |
| 7 | Jefferson_Airplane_(album) | 13.35 |
Title: Jefferson Airplane (album) Content: Jefferson Airplane is the eighth and final studio album by San Francisco rock band Jefferson Airplane, released on Epic Records in 1989. Both the album and the accompanying reunion tour featured the 1967-1970 lineup of the group, with the exception of former drummer Spencer Dryden. |
| 8 | Jefferson’s_Tree_of_Liberty | 13.34 |
Title: Jefferson’s Tree of Liberty Content: Jefferson’s Tree of Liberty is a 2008 studio album by Jefferson Starship, released on September 2, 2008. It is the tenth studio album recorded by the band under the Jefferson Starship name, the first studio album since 1999’s Windows of Heaven and the first on a major label by the band since 1989 when Starship’s Love Among the Cannibals and Jefferson Airplane’s reunion album, Jefferson Airplane were both released. |
| 9 | Jefferson_Airplane_Takes_Off | 13.21 |
Title: Jefferson Airplane Takes Off Content: Jefferson Airplane Takes Off is the debut album of San Francisco rock band Jefferson Airplane, released in August 1966 as RCA Victor LSP-3584 (stereo) & LPM-3584 (mono). The personnel differs from the later “classic” lineup and the music is more folk-rock than the harder psychedelic sound for which the band later became famous. Signe Toly Anderson was the female vocalist and Skip Spence played drums. |
| 10 | Mexico_(Jefferson_Airplane_song) | 13.19 |
Title: Mexico (Jefferson Airplane song) Content: “Mexico” is a single released in May 1970 by the San Francisco rock band Jefferson Airplane, produced by the band at Pacific High Recording Studios with Phill Sawyer as the recording engineer. |
| 11 | The_Roar_of_Jefferson_Airplane | 13.15 |
Title: The Roar of Jefferson Airplane Content: The Roar of Jefferson Airplane is a compilation of songs by San Francisco rock band Jefferson Airplane without the ubiquitous “White Rabbit”. “The Ballad of You and Me and Pooneil” is followed immediately by “The House at Pooneil Corners”, thus making a suite from the two similar and related songs originally released on separate albums. |
| 12 | Jefferson_Airplane_discography | 13.08 |
Title: Jefferson Airplane discography Content: The following is a comprehensive discography of Jefferson Airplane, an American rock band which formed in San Francisco in 1965. |
| 13 | Balince | 13.06 |
Title: Balince Content: Balince is a compilation album of Marty Balin’s work, including work from his solo albums and also including tracks from Jefferson Airplane, Jefferson Starship, and KBC Band. The album also includes unreleased tracks from the KBC Band sessions. The album was released shortly after the Jefferson Airplane reunion album and tour. |
| 14 | Cleared_for_Take_Off | 13.05 |
Title: Cleared for Take Off Content: Cleared For Take Off is a compilation album of music from the San Francisco rock band Jefferson Airplane.Much of the album features songs from the band’s first two albums, Jefferson Airplane Takes Off and Surrealistic Pillow. Other tracks like “Don’t Let Me Down” did not even appear on a Jefferson Airplane album. All of the tracks are live performances by the band. |
| 15 | KBC_Band | 13.02 |
Title: KBC Band Content: KBC Band was formed in 1985 by former Jefferson Airplane (later Jefferson Starship) members Paul Kantner (guitar and vocals), Marty Balin (vocals and guitar) and Jack Casady (bass). Other members included Keith Crossan (saxophone, guitar and vocals), Tim Gorman (keyboards and vocals), Mark “Slick” Aguilar (guitar and vocals) and Darrell Verdusco (drums). |
| 16 | White_Rabbit_(Jefferson_Airplane_song) | 12.68 |
Title: White Rabbit (Jefferson Airplane song) Content: “White Rabbit” is a song written by Grace Slick, and recorded by the American psychedelic rock band Jefferson Airplane for their 1967 album, Surrealistic Pillow. It was released as a single and became the band’s second top ten success, peaking at number eight on the Billboard Hot 100. |
| 17 | The_Peanut_Butter_Conspiracy | 12.27 |
Title: The Peanut Butter Conspiracy Content: The Peanut Butter Conspiracy was a Los Angeles-based psychedelic pop/rock group from the 1960s. The band is known for lead singer Barbara Robison, and for briefly having Spencer Dryden of Jefferson Airplane as a band member. |
| 18 | Signe_Toly_Anderson | 12.22 |
Title: Signe Toly Anderson Content: Signe Toly Anderson (born September 15, 1941) is an American singer who was one of the founding members of the American rock band Jefferson Airplane. |
| 19 | Jefferson_Airplane | 12.18 |
Title: Jefferson Airplane Content: Jefferson Airplane was an American rock band formed in San Francisco, California, in 1965. A pioneer of counterculture-era psychedelic rock, the group was the first band from the San Francisco scene to achieve international mainstream success. They performed at the three most famous American rock festivals of the 1960s—Monterey (1967), Woodstock (1969) and Altamont (1969)—as well as headlined the first Isle of Wight Festival (1968). |
| 20 | John_Barbata | 12.12 |
Title: John Barbata Content: John “Johny” Barbata (born April 1, 1945) is an American drummer, born in Passaic, New Jersey and raised in San Luis Obispo, California, active especially in pop and rock bands in the 1960s and 1970s, both as a band member and as a session drummer. Barbata has served as the drummer for The Turtles, Crosby, Stills, Nash & Young, Jefferson Airplane (for its final album and tour only), and Jefferson Starship (from 1974 to 1978). |
Reranking with the RankGPT prompt template
The RankGPT prompt template then provides those top-candidates in a sequence of messages, which end by providing the LLM the original search query, and asking it to rank them in order of their relevance:
Search Query: Members of the band Jefferson Airplane.
Rank the 20 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain."
The full /chat/completions payload
[
{
"role": "system",
"content": "You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query."
},
{
"role": "user",
"content": "I will provide you with 20 passages, each indicated by number identifier []. \nRank the passages based on their relevance to query: Members of the band Jefferson Airplane.."
},
{
"role": "assistant",
"content": "Okay, please provide the passages."
},
{
"role": "user",
"content": "[1] Title: Jefferson Airplane Loves You Content: Jefferson Airplane Loves You is a three-CD boxed set of recordings by the San Francisco rock band Jefferson Airplane with extensive liner notes by Jeff Tamarkin, author of the Jefferson Airplane history Got a Revolution: The Turbulent Flight of Jefferson Airplane.Many of the tracks are previously unreleased live recordings or studio rehearsals, but several are lifted from prior Jefferson Airplane albums."
},
{
"role": "assistant",
"content": "Received passage [1]."
},
{
"role": "user",
"content": "[2] Title: Through the Looking Glass (Jefferson Airplane album) Content: Through the Looking Glass is an Italian release of live performances by the San Francisco rock band Jefferson Airplane.The collection was originally issued as a two LP (long-playing vinyl) set on the Get Back label. The CD was released in 2002 on the Metrodome (DTK) label.Three of the tracks are credited to the \"Steel Riders\", a trio consisting of Jefferson Airplane band members Marty Balin, Jorma Kaukonen and Skip Spence."
},
{
"role": "assistant",
"content": "Received passage [2]."
},
{
"role": "user",
"content": "[3] Title: Jefferson Starship Content: Jefferson Starship is an American rock band formed in the early 1970s by several members of the former psychedelic rock group Jefferson Airplane. The band has undergone several major changes in personnel and genres through the years while retaining the same Jefferson Starship name."
},
{
"role": "assistant",
"content": "Received passage [3]."
},
{
"role": "user",
"content": "[4] Title: The Worst of Jefferson Airplane Content: The Worst of Jefferson Airplane is the first compilation album from the rock band Jefferson Airplane, released in November 1970 as RCA Victor LSP-4459. The album features all of Jefferson Airplane's hit singles up to that point. It peaked at #12 on the Billboard 200 in 1971 and has since gone platinum."
},
{
"role": "assistant",
"content": "Received passage [4]."
},
{
"role": "user",
"content": "[5] Title: Somebody to Love (Jefferson Airplane song) Content: \"Somebody to Love\", originally titled \"Someone to Love\", is a rock song that was written by Darby Slick, and originally recorded by 1960s folk rock band, The Great Society, and later by the psychedelic counterculture rock band Jefferson Airplane. Rolling Stone magazine ranked Jefferson Airplane's version No. 274 on their list of the 500 Greatest Songs of All Time."
},
{
"role": "assistant",
"content": "Received passage [5]."
},
{
"role": "user",
"content": "[6] Title: The Essential Jefferson Airplane Content: The Essential Jefferson Airplane is a compilation of music from the San Francisco rock band Jefferson Airplane spanning its entire career, excluding the brief reunion in 1989 (despite that album now being under common ownership with the rest of the band's catalog).It follows their development, in more-or-less chronological order, from their beginnings in folk-rock, through psychedelia, to conventional rock genres."
},
{
"role": "assistant",
"content": "Received passage [6]."
},
{
"role": "user",
"content": "[7] Title: Jefferson Airplane (album) Content: Jefferson Airplane is the eighth and final studio album by San Francisco rock band Jefferson Airplane, released on Epic Records in 1989. Both the album and the accompanying reunion tour featured the 1967-1970 lineup of the group, with the exception of former drummer Spencer Dryden."
},
{
"role": "assistant",
"content": "Received passage [7]."
},
{
"role": "user",
"content": "[8] Title: Jefferson's Tree of Liberty Content: Jefferson's Tree of Liberty is a 2008 studio album by Jefferson Starship, released on September 2, 2008. It is the tenth studio album recorded by the band under the Jefferson Starship name, the first studio album since 1999's Windows of Heaven and the first on a major label by the band since 1989 when Starship's Love Among the Cannibals and Jefferson Airplane's reunion album, Jefferson Airplane were both released."
},
{
"role": "assistant",
"content": "Received passage [8]."
},
{
"role": "user",
"content": "[9] Title: Jefferson Airplane Takes Off Content: Jefferson Airplane Takes Off is the debut album of San Francisco rock band Jefferson Airplane, released in August 1966 as RCA Victor LSP-3584 (stereo) & LPM-3584 (mono). The personnel differs from the later \"classic\" lineup and the music is more folk-rock than the harder psychedelic sound for which the band later became famous. Signe Toly Anderson was the female vocalist and Skip Spence played drums."
},
{
"role": "assistant",
"content": "Received passage [9]."
},
{
"role": "user",
"content": "[10] Title: Mexico (Jefferson Airplane song) Content: \"Mexico\" is a single released in May 1970 by the San Francisco rock band Jefferson Airplane, produced by the band at Pacific High Recording Studios with Phill Sawyer as the recording engineer."
},
{
"role": "assistant",
"content": "Received passage [10]."
},
{
"role": "user",
"content": "[11] Title: The Roar of Jefferson Airplane Content: The Roar of Jefferson Airplane is a compilation of songs by San Francisco rock band Jefferson Airplane without the ubiquitous \"White Rabbit\". \"The Ballad of You and Me and Pooneil\" is followed immediately by \"The House at Pooneil Corners\", thus making a suite from the two similar and related songs originally released on separate albums."
},
{
"role": "assistant",
"content": "Received passage [11]."
},
{
"role": "user",
"content": "[12] Title: Jefferson Airplane discography Content: The following is a comprehensive discography of Jefferson Airplane, an American rock band which formed in San Francisco in 1965."
},
{
"role": "assistant",
"content": "Received passage [12]."
},
{
"role": "user",
"content": "[13] Title: Balince Content: Balince is a compilation album of Marty Balin's work, including work from his solo albums and also including tracks from Jefferson Airplane, Jefferson Starship, and KBC Band. The album also includes unreleased tracks from the KBC Band sessions. The album was released shortly after the Jefferson Airplane reunion album and tour."
},
{
"role": "assistant",
"content": "Received passage [13]."
},
{
"role": "user",
"content": "[14] Title: Cleared for Take Off Content: Cleared For Take Off is a compilation album of music from the San Francisco rock band Jefferson Airplane.Much of the album features songs from the band's first two albums, Jefferson Airplane Takes Off and Surrealistic Pillow. Other tracks like \"Don't Let Me Down\" did not even appear on a Jefferson Airplane album. All of the tracks are live performances by the band."
},
{
"role": "assistant",
"content": "Received passage [14]."
},
{
"role": "user",
"content": "[15] Title: KBC Band Content: KBC Band was formed in 1985 by former Jefferson Airplane (later Jefferson Starship) members Paul Kantner (guitar and vocals), Marty Balin (vocals and guitar) and Jack Casady (bass). Other members included Keith Crossan (saxophone, guitar and vocals), Tim Gorman (keyboards and vocals), Mark \"Slick\" Aguilar (guitar and vocals) and Darrell Verdusco (drums)."
},
{
"role": "assistant",
"content": "Received passage [15]."
},
{
"role": "user",
"content": "[16] Title: White Rabbit (Jefferson Airplane song) Content: \"White Rabbit\" is a song written by Grace Slick, and recorded by the American psychedelic rock band Jefferson Airplane for their 1967 album, Surrealistic Pillow. It was released as a single and became the band's second top ten success, peaking at number eight on the Billboard Hot 100."
},
{
"role": "assistant",
"content": "Received passage [16]."
},
{
"role": "user",
"content": "[17] Title: The Peanut Butter Conspiracy Content: The Peanut Butter Conspiracy was a Los Angeles-based psychedelic pop/rock group from the 1960s. The band is known for lead singer Barbara Robison, and for briefly having Spencer Dryden of Jefferson Airplane as a band member."
},
{
"role": "assistant",
"content": "Received passage [17]."
},
{
"role": "user",
"content": "[18] Title: Signe Toly Anderson Content: Signe Toly Anderson (born September 15, 1941) is an American singer who was one of the founding members of the American rock band Jefferson Airplane."
},
{
"role": "assistant",
"content": "Received passage [18]."
},
{
"role": "user",
"content": "[19] Title: Jefferson Airplane Content: Jefferson Airplane was an American rock band formed in San Francisco, California, in 1965. A pioneer of counterculture-era psychedelic rock, the group was the first band from the San Francisco scene to achieve international mainstream success. They performed at the three most famous American rock festivals of the 1960s\u2014Monterey (1967), Woodstock (1969) and Altamont (1969)\u2014as well as headlined the first Isle of Wight Festival (1968)."
},
{
"role": "assistant",
"content": "Received passage [19]."
},
{
"role": "user",
"content": "[20] Title: John Barbata Content: John \"Johny\" Barbata (born April 1, 1945) is an American drummer, born in Passaic, New Jersey and raised in San Luis Obispo, California, active especially in pop and rock bands in the 1960s and 1970s, both as a band member and as a session drummer. Barbata has served as the drummer for The Turtles, Crosby, Stills, Nash & Young, Jefferson Airplane (for its final album and tour only), and Jefferson Starship (from 1974 to 1978)."
},
{
"role": "assistant",
"content": "Received passage [20]."
},
{
"role": "user",
"content": "Search Query: Members of the band Jefferson Airplane.. \nRank the 20 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain."
}
]The LLM’s response is a re-ordered list of documents, which then gets fed into the Blitzrank algorithm to determine which documents will be compared for reranking in the subsequent tournament round.
[15] KBC_Band > [18] Signe_Toly_Anderson > [20] John_Barbata > [9] Jefferson_Airplane_Takes_Off > [2] Through_the_Looking_Glass > [13] Balince > [17] The_Peanut_Butter_Conspiracy > [7] Jefferson_Airplane_(album) > [3] Jefferson_Starship > [19] Jefferson_Airplane > [12] Jefferson_Airplane_discography > [1] Jefferson_Airplane_Loves_You > [6] The_Essential_Jefferson_Airplane > [14] Cleared_for_Take_Off > [4] The_Worst_of_Jefferson_Airplane > [11] The_Roar_of_Jefferson_Airplane > [16] White_Rabbit_(Jefferson_Airplane_song) > [5] Somebody_to_Love_(Jefferson_Airplane_song) > [10] Mexico_(Jefferson_Airplane_song) > [8] Jefferson's_Tree_of_LibertyReranking with Jev pairwise comparisons
Typesafe’s Jev api, on the other hand, allows us to pack the query and each of those top-20 documents into the state, then ask Jev to make a pairwise choice about which document is most relevant to the query. For 20 candidates, we can literally ask Jev to compare each distinct pair, resulting in \(\binom{20}{2}=190\) total questions.
An aside: is 190 comparisons a bit excessive for just 20 candidates?
Probably. But, with our Jev api call, we’re just making explicit the comparisons that RankGPT is already doing implicitly. A few observations about this:
- The number of comparisons we ask Jev to make over a single
stateis just not a bottleneck in this pipeline. Sure, it costs us a bit more in input tokens and probably adds a bit of latency to the response, but those additions are relatively negligible. - I actually considered carrying out a mini-Blitzrank tournament among those top-20 candidates. The idea would be to first have Jev compare candidate 1 with candidate 2, etc, etc, then let Blitzrank determine how many comparisons to make. Interestingly though, Typesafe has a “parallel questions” blog post exploring a similar idea, which concludes that batching questions in parallel is “12.2x cheaper, 10.0x faster”. Along the dimensions of time and cost, it seems like the naive, exhaustive 190-question configuration is a decent default.
With 190 pairwise comparisons, it’s actually possible for Jev’s rankings to be inconsistent. It can say that a > b and b > c, but also that c > a. What’s neat is that Blitzrank represents this as a cycle in the graph and automatically handles this algorithmically as a need for further information.
Jev pairwise request and response — all 190 comparisons
Request
{
"model": "jev-latest",
"state": "Query: Members of the band Jefferson Airplane.\n\nDoc 1: Title: Jefferson Airplane Loves You Content: Jefferson Airplane Loves You is a three-CD boxed set of recordings by the San Francisco rock band Jefferson Airplane with extensive liner notes by Jeff Tamarkin, author of the Jefferson Airplane history Got a Revolution: The Turbulent Flight of Jefferson Airplane.Many of the tracks are previously unreleased live recordings or studio rehearsals, but several are lifted from prior Jefferson Airplane albums.\n\nDoc 2: Title: Through the Looking Glass (Jefferson Airplane album) Content: Through the Looking Glass is an Italian release of live performances by the San Francisco rock band Jefferson Airplane.The collection was originally issued as a two LP (long-playing vinyl) set on the Get Back label. The CD was released in 2002 on the Metrodome (DTK) label.Three of the tracks are credited to the \"Steel Riders\", a trio consisting of Jefferson Airplane band members Marty Balin, Jorma Kaukonen and Skip Spence.\n\nDoc 3: Title: Jefferson Starship Content: Jefferson Starship is an American rock band formed in the early 1970s by several members of the former psychedelic rock group Jefferson Airplane. The band has undergone several major changes in personnel and genres through the years while retaining the same Jefferson Starship name.\n\nDoc 4: Title: The Worst of Jefferson Airplane Content: The Worst of Jefferson Airplane is the first compilation album from the rock band Jefferson Airplane, released in November 1970 as RCA Victor LSP-4459. The album features all of Jefferson Airplane's hit singles up to that point. It peaked at #12 on the Billboard 200 in 1971 and has since gone platinum.\n\nDoc 5: Title: Somebody to Love (Jefferson Airplane song) Content: \"Somebody to Love\", originally titled \"Someone to Love\", is a rock song that was written by Darby Slick, and originally recorded by 1960s folk rock band, The Great Society, and later by the psychedelic counterculture rock band Jefferson Airplane. Rolling Stone magazine ranked Jefferson Airplane's version No. 274 on their list of the 500 Greatest Songs of All Time.\n\nDoc 6: Title: The Essential Jefferson Airplane Content: The Essential Jefferson Airplane is a compilation of music from the San Francisco rock band Jefferson Airplane spanning its entire career, excluding the brief reunion in 1989 (despite that album now being under common ownership with the rest of the band's catalog).It follows their development, in more-or-less chronological order, from their beginnings in folk-rock, through psychedelia, to conventional rock genres.\n\nDoc 7: Title: Jefferson Airplane (album) Content: Jefferson Airplane is the eighth and final studio album by San Francisco rock band Jefferson Airplane, released on Epic Records in 1989. Both the album and the accompanying reunion tour featured the 1967-1970 lineup of the group, with the exception of former drummer Spencer Dryden.\n\nDoc 8: Title: Jefferson's Tree of Liberty Content: Jefferson's Tree of Liberty is a 2008 studio album by Jefferson Starship, released on September 2, 2008. It is the tenth studio album recorded by the band under the Jefferson Starship name, the first studio album since 1999's Windows of Heaven and the first on a major label by the band since 1989 when Starship's Love Among the Cannibals and Jefferson Airplane's reunion album, Jefferson Airplane were both released.\n\nDoc 9: Title: Jefferson Airplane Takes Off Content: Jefferson Airplane Takes Off is the debut album of San Francisco rock band Jefferson Airplane, released in August 1966 as RCA Victor LSP-3584 (stereo) & LPM-3584 (mono). The personnel differs from the later \"classic\" lineup and the music is more folk-rock than the harder psychedelic sound for which the band later became famous. Signe Toly Anderson was the female vocalist and Skip Spence played drums.\n\nDoc 10: Title: Mexico (Jefferson Airplane song) Content: \"Mexico\" is a single released in May 1970 by the San Francisco rock band Jefferson Airplane, produced by the band at Pacific High Recording Studios with Phill Sawyer as the recording engineer.\n\nDoc 11: Title: The Roar of Jefferson Airplane Content: The Roar of Jefferson Airplane is a compilation of songs by San Francisco rock band Jefferson Airplane without the ubiquitous \"White Rabbit\". \"The Ballad of You and Me and Pooneil\" is followed immediately by \"The House at Pooneil Corners\", thus making a suite from the two similar and related songs originally released on separate albums.\n\nDoc 12: Title: Jefferson Airplane discography Content: The following is a comprehensive discography of Jefferson Airplane, an American rock band which formed in San Francisco in 1965.\n\nDoc 13: Title: Balince Content: Balince is a compilation album of Marty Balin's work, including work from his solo albums and also including tracks from Jefferson Airplane, Jefferson Starship, and KBC Band. The album also includes unreleased tracks from the KBC Band sessions. The album was released shortly after the Jefferson Airplane reunion album and tour.\n\nDoc 14: Title: Cleared for Take Off Content: Cleared For Take Off is a compilation album of music from the San Francisco rock band Jefferson Airplane.Much of the album features songs from the band's first two albums, Jefferson Airplane Takes Off and Surrealistic Pillow. Other tracks like \"Don't Let Me Down\" did not even appear on a Jefferson Airplane album. All of the tracks are live performances by the band.\n\nDoc 15: Title: KBC Band Content: KBC Band was formed in 1985 by former Jefferson Airplane (later Jefferson Starship) members Paul Kantner (guitar and vocals), Marty Balin (vocals and guitar) and Jack Casady (bass). Other members included Keith Crossan (saxophone, guitar and vocals), Tim Gorman (keyboards and vocals), Mark \"Slick\" Aguilar (guitar and vocals) and Darrell Verdusco (drums).\n\nDoc 16: Title: White Rabbit (Jefferson Airplane song) Content: \"White Rabbit\" is a song written by Grace Slick, and recorded by the American psychedelic rock band Jefferson Airplane for their 1967 album, Surrealistic Pillow. It was released as a single and became the band's second top ten success, peaking at number eight on the Billboard Hot 100.\n\nDoc 17: Title: The Peanut Butter Conspiracy Content: The Peanut Butter Conspiracy was a Los Angeles-based psychedelic pop/rock group from the 1960s. The band is known for lead singer Barbara Robison, and for briefly having Spencer Dryden of Jefferson Airplane as a band member.\n\nDoc 18: Title: Signe Toly Anderson Content: Signe Toly Anderson (born September 15, 1941) is an American singer who was one of the founding members of the American rock band Jefferson Airplane.\n\nDoc 19: Title: Jefferson Airplane Content: Jefferson Airplane was an American rock band formed in San Francisco, California, in 1965. A pioneer of counterculture-era psychedelic rock, the group was the first band from the San Francisco scene to achieve international mainstream success. They performed at the three most famous American rock festivals of the 1960s\u2014Monterey (1967), Woodstock (1969) and Altamont (1969)\u2014as well as headlined the first Isle of Wight Festival (1968).\n\nDoc 20: Title: John Barbata Content: John \"Johny\" Barbata (born April 1, 1945) is an American drummer, born in Passaic, New Jersey and raised in San Luis Obispo, California, active especially in pop and rock bands in the 1960s and 1970s, both as a band member and as a session drummer. Barbata has served as the drummer for The Turtles, Crosby, Stills, Nash & Young, Jefferson Airplane (for its final album and tour only), and Jefferson Starship (from 1974 to 1978).",
"questions": {
"pair_1_2": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 2 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 2"
}
},
"pair_1_3": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 3 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 3"
}
},
"pair_1_4": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 4 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 4"
}
},
"pair_1_5": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 5 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 5"
}
},
"pair_1_6": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 6 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 6"
}
},
"pair_1_7": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 7 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 7"
}
},
"pair_1_8": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 8 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 8"
}
},
"pair_1_9": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 9 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 9"
}
},
"pair_1_10": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 10"
}
},
"pair_1_11": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 11"
}
},
"pair_1_12": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 12"
}
},
"pair_1_13": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 13"
}
},
"pair_1_14": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 14"
}
},
"pair_1_15": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 15"
}
},
"pair_1_16": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 16"
}
},
"pair_1_17": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 17"
}
},
"pair_1_18": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 18"
}
},
"pair_1_19": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 19"
}
},
"pair_1_20": {
"type": "choice",
"instructions": "Between Doc 1 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 1",
"right": "Doc 20"
}
},
"pair_2_3": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 3 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 3"
}
},
"pair_2_4": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 4 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 4"
}
},
"pair_2_5": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 5 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 5"
}
},
"pair_2_6": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 6 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 6"
}
},
"pair_2_7": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 7 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 7"
}
},
"pair_2_8": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 8 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 8"
}
},
"pair_2_9": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 9 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 9"
}
},
"pair_2_10": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 10"
}
},
"pair_2_11": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 11"
}
},
"pair_2_12": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 12"
}
},
"pair_2_13": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 13"
}
},
"pair_2_14": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 14"
}
},
"pair_2_15": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 15"
}
},
"pair_2_16": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 16"
}
},
"pair_2_17": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 17"
}
},
"pair_2_18": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 18"
}
},
"pair_2_19": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 19"
}
},
"pair_2_20": {
"type": "choice",
"instructions": "Between Doc 2 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 2",
"right": "Doc 20"
}
},
"pair_3_4": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 4 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 4"
}
},
"pair_3_5": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 5 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 5"
}
},
"pair_3_6": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 6 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 6"
}
},
"pair_3_7": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 7 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 7"
}
},
"pair_3_8": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 8 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 8"
}
},
"pair_3_9": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 9 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 9"
}
},
"pair_3_10": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 10"
}
},
"pair_3_11": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 11"
}
},
"pair_3_12": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 12"
}
},
"pair_3_13": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 13"
}
},
"pair_3_14": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 14"
}
},
"pair_3_15": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 15"
}
},
"pair_3_16": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 16"
}
},
"pair_3_17": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 17"
}
},
"pair_3_18": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 18"
}
},
"pair_3_19": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 19"
}
},
"pair_3_20": {
"type": "choice",
"instructions": "Between Doc 3 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 3",
"right": "Doc 20"
}
},
"pair_4_5": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 5 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 5"
}
},
"pair_4_6": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 6 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 6"
}
},
"pair_4_7": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 7 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 7"
}
},
"pair_4_8": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 8 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 8"
}
},
"pair_4_9": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 9 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 9"
}
},
"pair_4_10": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 10"
}
},
"pair_4_11": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 11"
}
},
"pair_4_12": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 12"
}
},
"pair_4_13": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 13"
}
},
"pair_4_14": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 14"
}
},
"pair_4_15": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 15"
}
},
"pair_4_16": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 16"
}
},
"pair_4_17": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 17"
}
},
"pair_4_18": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 18"
}
},
"pair_4_19": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 19"
}
},
"pair_4_20": {
"type": "choice",
"instructions": "Between Doc 4 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 4",
"right": "Doc 20"
}
},
"pair_5_6": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 6 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 6"
}
},
"pair_5_7": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 7 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 7"
}
},
"pair_5_8": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 8 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 8"
}
},
"pair_5_9": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 9 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 9"
}
},
"pair_5_10": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 10"
}
},
"pair_5_11": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 11"
}
},
"pair_5_12": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 12"
}
},
"pair_5_13": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 13"
}
},
"pair_5_14": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 14"
}
},
"pair_5_15": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 15"
}
},
"pair_5_16": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 16"
}
},
"pair_5_17": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 17"
}
},
"pair_5_18": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 18"
}
},
"pair_5_19": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 19"
}
},
"pair_5_20": {
"type": "choice",
"instructions": "Between Doc 5 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 5",
"right": "Doc 20"
}
},
"pair_6_7": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 7 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 7"
}
},
"pair_6_8": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 8 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 8"
}
},
"pair_6_9": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 9 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 9"
}
},
"pair_6_10": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 10"
}
},
"pair_6_11": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 11"
}
},
"pair_6_12": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 12"
}
},
"pair_6_13": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 13"
}
},
"pair_6_14": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 14"
}
},
"pair_6_15": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 15"
}
},
"pair_6_16": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 16"
}
},
"pair_6_17": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 17"
}
},
"pair_6_18": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 18"
}
},
"pair_6_19": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 19"
}
},
"pair_6_20": {
"type": "choice",
"instructions": "Between Doc 6 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 6",
"right": "Doc 20"
}
},
"pair_7_8": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 8 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 8"
}
},
"pair_7_9": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 9 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 9"
}
},
"pair_7_10": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 10"
}
},
"pair_7_11": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 11"
}
},
"pair_7_12": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 12"
}
},
"pair_7_13": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 13"
}
},
"pair_7_14": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 14"
}
},
"pair_7_15": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 15"
}
},
"pair_7_16": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 16"
}
},
"pair_7_17": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 17"
}
},
"pair_7_18": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 18"
}
},
"pair_7_19": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 19"
}
},
"pair_7_20": {
"type": "choice",
"instructions": "Between Doc 7 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 7",
"right": "Doc 20"
}
},
"pair_8_9": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 9 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 9"
}
},
"pair_8_10": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 10"
}
},
"pair_8_11": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 11"
}
},
"pair_8_12": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 12"
}
},
"pair_8_13": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 13"
}
},
"pair_8_14": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 14"
}
},
"pair_8_15": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 15"
}
},
"pair_8_16": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 16"
}
},
"pair_8_17": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 17"
}
},
"pair_8_18": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 18"
}
},
"pair_8_19": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 19"
}
},
"pair_8_20": {
"type": "choice",
"instructions": "Between Doc 8 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 8",
"right": "Doc 20"
}
},
"pair_9_10": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 10 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 10"
}
},
"pair_9_11": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 11"
}
},
"pair_9_12": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 12"
}
},
"pair_9_13": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 13"
}
},
"pair_9_14": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 14"
}
},
"pair_9_15": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 15"
}
},
"pair_9_16": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 16"
}
},
"pair_9_17": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 17"
}
},
"pair_9_18": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 18"
}
},
"pair_9_19": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 19"
}
},
"pair_9_20": {
"type": "choice",
"instructions": "Between Doc 9 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 9",
"right": "Doc 20"
}
},
"pair_10_11": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 11 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 11"
}
},
"pair_10_12": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 12"
}
},
"pair_10_13": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 13"
}
},
"pair_10_14": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 14"
}
},
"pair_10_15": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 15"
}
},
"pair_10_16": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 16"
}
},
"pair_10_17": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 17"
}
},
"pair_10_18": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 18"
}
},
"pair_10_19": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 19"
}
},
"pair_10_20": {
"type": "choice",
"instructions": "Between Doc 10 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 10",
"right": "Doc 20"
}
},
"pair_11_12": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 12 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 12"
}
},
"pair_11_13": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 13"
}
},
"pair_11_14": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 14"
}
},
"pair_11_15": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 15"
}
},
"pair_11_16": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 16"
}
},
"pair_11_17": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 17"
}
},
"pair_11_18": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 18"
}
},
"pair_11_19": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 19"
}
},
"pair_11_20": {
"type": "choice",
"instructions": "Between Doc 11 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 11",
"right": "Doc 20"
}
},
"pair_12_13": {
"type": "choice",
"instructions": "Between Doc 12 and Doc 13 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 12",
"right": "Doc 13"
}
},
"pair_12_14": {
"type": "choice",
"instructions": "Between Doc 12 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 12",
"right": "Doc 14"
}
},
"pair_12_15": {
"type": "choice",
"instructions": "Between Doc 12 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 12",
"right": "Doc 15"
}
},
"pair_12_16": {
"type": "choice",
"instructions": "Between Doc 12 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 12",
"right": "Doc 16"
}
},
"pair_12_17": {
"type": "choice",
"instructions": "Between Doc 12 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 12",
"right": "Doc 17"
}
},
"pair_12_18": {
"type": "choice",
"instructions": "Between Doc 12 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 12",
"right": "Doc 18"
}
},
"pair_12_19": {
"type": "choice",
"instructions": "Between Doc 12 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 12",
"right": "Doc 19"
}
},
"pair_12_20": {
"type": "choice",
"instructions": "Between Doc 12 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 12",
"right": "Doc 20"
}
},
"pair_13_14": {
"type": "choice",
"instructions": "Between Doc 13 and Doc 14 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 13",
"right": "Doc 14"
}
},
"pair_13_15": {
"type": "choice",
"instructions": "Between Doc 13 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 13",
"right": "Doc 15"
}
},
"pair_13_16": {
"type": "choice",
"instructions": "Between Doc 13 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 13",
"right": "Doc 16"
}
},
"pair_13_17": {
"type": "choice",
"instructions": "Between Doc 13 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 13",
"right": "Doc 17"
}
},
"pair_13_18": {
"type": "choice",
"instructions": "Between Doc 13 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 13",
"right": "Doc 18"
}
},
"pair_13_19": {
"type": "choice",
"instructions": "Between Doc 13 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 13",
"right": "Doc 19"
}
},
"pair_13_20": {
"type": "choice",
"instructions": "Between Doc 13 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 13",
"right": "Doc 20"
}
},
"pair_14_15": {
"type": "choice",
"instructions": "Between Doc 14 and Doc 15 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 14",
"right": "Doc 15"
}
},
"pair_14_16": {
"type": "choice",
"instructions": "Between Doc 14 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 14",
"right": "Doc 16"
}
},
"pair_14_17": {
"type": "choice",
"instructions": "Between Doc 14 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 14",
"right": "Doc 17"
}
},
"pair_14_18": {
"type": "choice",
"instructions": "Between Doc 14 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 14",
"right": "Doc 18"
}
},
"pair_14_19": {
"type": "choice",
"instructions": "Between Doc 14 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 14",
"right": "Doc 19"
}
},
"pair_14_20": {
"type": "choice",
"instructions": "Between Doc 14 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 14",
"right": "Doc 20"
}
},
"pair_15_16": {
"type": "choice",
"instructions": "Between Doc 15 and Doc 16 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 15",
"right": "Doc 16"
}
},
"pair_15_17": {
"type": "choice",
"instructions": "Between Doc 15 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 15",
"right": "Doc 17"
}
},
"pair_15_18": {
"type": "choice",
"instructions": "Between Doc 15 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 15",
"right": "Doc 18"
}
},
"pair_15_19": {
"type": "choice",
"instructions": "Between Doc 15 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 15",
"right": "Doc 19"
}
},
"pair_15_20": {
"type": "choice",
"instructions": "Between Doc 15 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 15",
"right": "Doc 20"
}
},
"pair_16_17": {
"type": "choice",
"instructions": "Between Doc 16 and Doc 17 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 16",
"right": "Doc 17"
}
},
"pair_16_18": {
"type": "choice",
"instructions": "Between Doc 16 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 16",
"right": "Doc 18"
}
},
"pair_16_19": {
"type": "choice",
"instructions": "Between Doc 16 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 16",
"right": "Doc 19"
}
},
"pair_16_20": {
"type": "choice",
"instructions": "Between Doc 16 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 16",
"right": "Doc 20"
}
},
"pair_17_18": {
"type": "choice",
"instructions": "Between Doc 17 and Doc 18 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 17",
"right": "Doc 18"
}
},
"pair_17_19": {
"type": "choice",
"instructions": "Between Doc 17 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 17",
"right": "Doc 19"
}
},
"pair_17_20": {
"type": "choice",
"instructions": "Between Doc 17 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 17",
"right": "Doc 20"
}
},
"pair_18_19": {
"type": "choice",
"instructions": "Between Doc 18 and Doc 19 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 18",
"right": "Doc 19"
}
},
"pair_18_20": {
"type": "choice",
"instructions": "Between Doc 18 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 18",
"right": "Doc 20"
}
},
"pair_19_20": {
"type": "choice",
"instructions": "Between Doc 19 and Doc 20 (both shown in the state), which doc is more relevant to the query?",
"criteria": {
"left": "Doc 19",
"right": "Doc 20"
}
}
}
}Response
{
"model": "jev-1.14.0",
"answers": {
"pair_1_2": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_1_3": {
"type": "choice",
"choice": "right",
"confidence": 0.64,
"probabilities": {
"left": 0.18,
"right": 0.82
}
},
"pair_1_4": {
"type": "choice",
"choice": "right",
"confidence": 0.26,
"probabilities": {
"left": 0.37,
"right": 0.63
}
},
"pair_1_5": {
"type": "choice",
"choice": "right",
"confidence": 0.48,
"probabilities": {
"left": 0.26,
"right": 0.74
}
},
"pair_1_6": {
"type": "choice",
"choice": "right",
"confidence": 0.54,
"probabilities": {
"left": 0.23,
"right": 0.77
}
},
"pair_1_7": {
"type": "choice",
"choice": "right",
"confidence": 0.98,
"probabilities": {
"left": 0.01,
"right": 0.99
}
},
"pair_1_8": {
"type": "choice",
"choice": "left",
"confidence": 0.44,
"probabilities": {
"left": 0.72,
"right": 0.28
}
},
"pair_1_9": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_1_10": {
"type": "choice",
"choice": "right",
"confidence": 0.36,
"probabilities": {
"left": 0.32,
"right": 0.68
}
},
"pair_1_11": {
"type": "choice",
"choice": "left",
"confidence": 0.04,
"probabilities": {
"left": 0.52,
"right": 0.48
}
},
"pair_1_12": {
"type": "choice",
"choice": "right",
"confidence": 0.82,
"probabilities": {
"left": 0.09,
"right": 0.91
}
},
"pair_1_13": {
"type": "choice",
"choice": "right",
"confidence": 0.22,
"probabilities": {
"left": 0.39,
"right": 0.61
}
},
"pair_1_14": {
"type": "choice",
"choice": "right",
"confidence": 0.28,
"probabilities": {
"left": 0.36,
"right": 0.64
}
},
"pair_1_15": {
"type": "choice",
"choice": "right",
"confidence": 0.9,
"probabilities": {
"left": 0.05,
"right": 0.95
}
},
"pair_1_16": {
"type": "choice",
"choice": "right",
"confidence": 0.76,
"probabilities": {
"left": 0.12,
"right": 0.88
}
},
"pair_1_17": {
"type": "choice",
"choice": "right",
"confidence": 0.64,
"probabilities": {
"left": 0.18,
"right": 0.82
}
},
"pair_1_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_1_19": {
"type": "choice",
"choice": "right",
"confidence": 0.86,
"probabilities": {
"left": 0.07,
"right": 0.93
}
},
"pair_1_20": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_2_3": {
"type": "choice",
"choice": "left",
"confidence": 0.4,
"probabilities": {
"left": 0.7,
"right": 0.3
}
},
"pair_2_4": {
"type": "choice",
"choice": "left",
"confidence": 0.9,
"probabilities": {
"left": 0.95,
"right": 0.05
}
},
"pair_2_5": {
"type": "choice",
"choice": "left",
"confidence": 0.9,
"probabilities": {
"left": 0.95,
"right": 0.05
}
},
"pair_2_6": {
"type": "choice",
"choice": "left",
"confidence": 0.68,
"probabilities": {
"left": 0.84,
"right": 0.16
}
},
"pair_2_7": {
"type": "choice",
"choice": "right",
"confidence": 0.08,
"probabilities": {
"left": 0.46,
"right": 0.54
}
},
"pair_2_8": {
"type": "choice",
"choice": "left",
"confidence": 0.98,
"probabilities": {
"left": 0.99,
"right": 0.01
}
},
"pair_2_9": {
"type": "choice",
"choice": "right",
"confidence": 0.76,
"probabilities": {
"left": 0.12,
"right": 0.88
}
},
"pair_2_10": {
"type": "choice",
"choice": "left",
"confidence": 0.96,
"probabilities": {
"left": 0.98,
"right": 0.02
}
},
"pair_2_11": {
"type": "choice",
"choice": "left",
"confidence": 0.98,
"probabilities": {
"left": 0.99,
"right": 0.01
}
},
"pair_2_12": {
"type": "choice",
"choice": "left",
"confidence": 0.06,
"probabilities": {
"left": 0.53,
"right": 0.47
}
},
"pair_2_13": {
"type": "choice",
"choice": "left",
"confidence": 0.9,
"probabilities": {
"left": 0.95,
"right": 0.05
}
},
"pair_2_14": {
"type": "choice",
"choice": "left",
"confidence": 0.9,
"probabilities": {
"left": 0.95,
"right": 0.05
}
},
"pair_2_15": {
"type": "choice",
"choice": "left",
"confidence": 0.08,
"probabilities": {
"left": 0.54,
"right": 0.46
}
},
"pair_2_16": {
"type": "choice",
"choice": "left",
"confidence": 0.6,
"probabilities": {
"left": 0.8,
"right": 0.2
}
},
"pair_2_17": {
"type": "choice",
"choice": "left",
"confidence": 0.82,
"probabilities": {
"left": 0.91,
"right": 0.09
}
},
"pair_2_18": {
"type": "choice",
"choice": "right",
"confidence": 0.88,
"probabilities": {
"left": 0.06,
"right": 0.94
}
},
"pair_2_19": {
"type": "choice",
"choice": "left",
"confidence": 0.36,
"probabilities": {
"left": 0.68,
"right": 0.32
}
},
"pair_2_20": {
"type": "choice",
"choice": "right",
"confidence": 0.06,
"probabilities": {
"left": 0.47,
"right": 0.53
}
},
"pair_3_4": {
"type": "choice",
"choice": "left",
"confidence": 0.6,
"probabilities": {
"left": 0.8,
"right": 0.2
}
},
"pair_3_5": {
"type": "choice",
"choice": "left",
"confidence": 0.5,
"probabilities": {
"left": 0.75,
"right": 0.25
}
},
"pair_3_6": {
"type": "choice",
"choice": "left",
"confidence": 0.36,
"probabilities": {
"left": 0.68,
"right": 0.32
}
},
"pair_3_7": {
"type": "choice",
"choice": "right",
"confidence": 0.8,
"probabilities": {
"left": 0.1,
"right": 0.9
}
},
"pair_3_8": {
"type": "choice",
"choice": "left",
"confidence": 0.94,
"probabilities": {
"left": 0.97,
"right": 0.03
}
},
"pair_3_9": {
"type": "choice",
"choice": "right",
"confidence": 0.94,
"probabilities": {
"left": 0.03,
"right": 0.97
}
},
"pair_3_10": {
"type": "choice",
"choice": "left",
"confidence": 0.74,
"probabilities": {
"left": 0.87,
"right": 0.13
}
},
"pair_3_11": {
"type": "choice",
"choice": "left",
"confidence": 0.66,
"probabilities": {
"left": 0.83,
"right": 0.17
}
},
"pair_3_12": {
"type": "choice",
"choice": "right",
"confidence": 0.12,
"probabilities": {
"left": 0.44,
"right": 0.56
}
},
"pair_3_13": {
"type": "choice",
"choice": "left",
"confidence": 0.58,
"probabilities": {
"left": 0.79,
"right": 0.21
}
},
"pair_3_14": {
"type": "choice",
"choice": "left",
"confidence": 0.44,
"probabilities": {
"left": 0.72,
"right": 0.28
}
},
"pair_3_15": {
"type": "choice",
"choice": "right",
"confidence": 0.38,
"probabilities": {
"left": 0.31,
"right": 0.69
}
},
"pair_3_16": {
"type": "choice",
"choice": "right",
"confidence": 0.18,
"probabilities": {
"left": 0.41,
"right": 0.59
}
},
"pair_3_17": {
"type": "choice",
"choice": "left",
"confidence": 0.38,
"probabilities": {
"left": 0.69,
"right": 0.31
}
},
"pair_3_18": {
"type": "choice",
"choice": "right",
"confidence": 0.98,
"probabilities": {
"left": 0.01,
"right": 0.99
}
},
"pair_3_19": {
"type": "choice",
"choice": "right",
"confidence": 0.16,
"probabilities": {
"left": 0.42,
"right": 0.58
}
},
"pair_3_20": {
"type": "choice",
"choice": "right",
"confidence": 0.68,
"probabilities": {
"left": 0.16,
"right": 0.84
}
},
"pair_4_5": {
"type": "choice",
"choice": "right",
"confidence": 0.48,
"probabilities": {
"left": 0.26,
"right": 0.74
}
},
"pair_4_6": {
"type": "choice",
"choice": "right",
"confidence": 0.6,
"probabilities": {
"left": 0.2,
"right": 0.8
}
},
"pair_4_7": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_4_8": {
"type": "choice",
"choice": "left",
"confidence": 0.58,
"probabilities": {
"left": 0.79,
"right": 0.21
}
},
"pair_4_9": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_4_10": {
"type": "choice",
"choice": "right",
"confidence": 0.14,
"probabilities": {
"left": 0.43,
"right": 0.57
}
},
"pair_4_11": {
"type": "choice",
"choice": "left",
"confidence": 0.16,
"probabilities": {
"left": 0.58,
"right": 0.42
}
},
"pair_4_12": {
"type": "choice",
"choice": "right",
"confidence": 0.78,
"probabilities": {
"left": 0.11,
"right": 0.89
}
},
"pair_4_13": {
"type": "choice",
"choice": "right",
"confidence": 0.38,
"probabilities": {
"left": 0.31,
"right": 0.69
}
},
"pair_4_14": {
"type": "choice",
"choice": "right",
"confidence": 0.16,
"probabilities": {
"left": 0.42,
"right": 0.58
}
},
"pair_4_15": {
"type": "choice",
"choice": "right",
"confidence": 0.82,
"probabilities": {
"left": 0.09,
"right": 0.91
}
},
"pair_4_16": {
"type": "choice",
"choice": "right",
"confidence": 0.86,
"probabilities": {
"left": 0.07,
"right": 0.93
}
},
"pair_4_17": {
"type": "choice",
"choice": "right",
"confidence": 0.56,
"probabilities": {
"left": 0.22,
"right": 0.78
}
},
"pair_4_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_4_19": {
"type": "choice",
"choice": "right",
"confidence": 0.9,
"probabilities": {
"left": 0.05,
"right": 0.95
}
},
"pair_4_20": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_5_6": {
"type": "choice",
"choice": "right",
"confidence": 0.7,
"probabilities": {
"left": 0.15,
"right": 0.85
}
},
"pair_5_7": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_5_8": {
"type": "choice",
"choice": "left",
"confidence": 0.48,
"probabilities": {
"left": 0.74,
"right": 0.26
}
},
"pair_5_9": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_5_10": {
"type": "choice",
"choice": "left",
"confidence": 0.16,
"probabilities": {
"left": 0.58,
"right": 0.42
}
},
"pair_5_11": {
"type": "choice",
"choice": "right",
"confidence": 0.02,
"probabilities": {
"left": 0.49,
"right": 0.51
}
},
"pair_5_12": {
"type": "choice",
"choice": "right",
"confidence": 0.72,
"probabilities": {
"left": 0.14,
"right": 0.86
}
},
"pair_5_13": {
"type": "choice",
"choice": "right",
"confidence": 0.24,
"probabilities": {
"left": 0.38,
"right": 0.62
}
},
"pair_5_14": {
"type": "choice",
"choice": "right",
"confidence": 0.28,
"probabilities": {
"left": 0.36,
"right": 0.64
}
},
"pair_5_15": {
"type": "choice",
"choice": "right",
"confidence": 0.84,
"probabilities": {
"left": 0.08,
"right": 0.92
}
},
"pair_5_16": {
"type": "choice",
"choice": "right",
"confidence": 0.6,
"probabilities": {
"left": 0.2,
"right": 0.8
}
},
"pair_5_17": {
"type": "choice",
"choice": "right",
"confidence": 0.36,
"probabilities": {
"left": 0.32,
"right": 0.68
}
},
"pair_5_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_5_19": {
"type": "choice",
"choice": "right",
"confidence": 0.8,
"probabilities": {
"left": 0.1,
"right": 0.9
}
},
"pair_5_20": {
"type": "choice",
"choice": "right",
"confidence": 0.94,
"probabilities": {
"left": 0.03,
"right": 0.97
}
},
"pair_6_7": {
"type": "choice",
"choice": "right",
"confidence": 0.92,
"probabilities": {
"left": 0.04,
"right": 0.96
}
},
"pair_6_8": {
"type": "choice",
"choice": "left",
"confidence": 0.78,
"probabilities": {
"left": 0.89,
"right": 0.11
}
},
"pair_6_9": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_6_10": {
"type": "choice",
"choice": "left",
"confidence": 0.12,
"probabilities": {
"left": 0.56,
"right": 0.44
}
},
"pair_6_11": {
"type": "choice",
"choice": "left",
"confidence": 0.46,
"probabilities": {
"left": 0.73,
"right": 0.27
}
},
"pair_6_12": {
"type": "choice",
"choice": "right",
"confidence": 0.74,
"probabilities": {
"left": 0.13,
"right": 0.87
}
},
"pair_6_13": {
"type": "choice",
"choice": "left",
"confidence": 0.44,
"probabilities": {
"left": 0.72,
"right": 0.28
}
},
"pair_6_14": {
"type": "choice",
"choice": "left",
"confidence": 0.18,
"probabilities": {
"left": 0.59,
"right": 0.41
}
},
"pair_6_15": {
"type": "choice",
"choice": "right",
"confidence": 0.56,
"probabilities": {
"left": 0.22,
"right": 0.78
}
},
"pair_6_16": {
"type": "choice",
"choice": "right",
"confidence": 0.64,
"probabilities": {
"left": 0.18,
"right": 0.82
}
},
"pair_6_17": {
"type": "choice",
"choice": "left",
"confidence": 0.08,
"probabilities": {
"left": 0.54,
"right": 0.46
}
},
"pair_6_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_6_19": {
"type": "choice",
"choice": "right",
"confidence": 0.78,
"probabilities": {
"left": 0.11,
"right": 0.89
}
},
"pair_6_20": {
"type": "choice",
"choice": "right",
"confidence": 0.9,
"probabilities": {
"left": 0.05,
"right": 0.95
}
},
"pair_7_8": {
"type": "choice",
"choice": "left",
"confidence": 0.98,
"probabilities": {
"left": 0.99,
"right": 0.01
}
},
"pair_7_9": {
"type": "choice",
"choice": "right",
"confidence": 0.54,
"probabilities": {
"left": 0.23,
"right": 0.77
}
},
"pair_7_10": {
"type": "choice",
"choice": "left",
"confidence": 0.94,
"probabilities": {
"left": 0.97,
"right": 0.03
}
},
"pair_7_11": {
"type": "choice",
"choice": "left",
"confidence": 0.96,
"probabilities": {
"left": 0.98,
"right": 0.02
}
},
"pair_7_12": {
"type": "choice",
"choice": "left",
"confidence": 0.28,
"probabilities": {
"left": 0.64,
"right": 0.36
}
},
"pair_7_13": {
"type": "choice",
"choice": "left",
"confidence": 0.9,
"probabilities": {
"left": 0.95,
"right": 0.05
}
},
"pair_7_14": {
"type": "choice",
"choice": "left",
"confidence": 0.9,
"probabilities": {
"left": 0.95,
"right": 0.05
}
},
"pair_7_15": {
"type": "choice",
"choice": "left",
"confidence": 0.54,
"probabilities": {
"left": 0.77,
"right": 0.23
}
},
"pair_7_16": {
"type": "choice",
"choice": "left",
"confidence": 0.38,
"probabilities": {
"left": 0.69,
"right": 0.31
}
},
"pair_7_17": {
"type": "choice",
"choice": "left",
"confidence": 0.88,
"probabilities": {
"left": 0.94,
"right": 0.06
}
},
"pair_7_18": {
"type": "choice",
"choice": "right",
"confidence": 0.86,
"probabilities": {
"left": 0.07,
"right": 0.93
}
},
"pair_7_19": {
"type": "choice",
"choice": "left",
"confidence": 0.44,
"probabilities": {
"left": 0.72,
"right": 0.28
}
},
"pair_7_20": {
"type": "choice",
"choice": "right",
"confidence": 0.18,
"probabilities": {
"left": 0.41,
"right": 0.59
}
},
"pair_8_9": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_8_10": {
"type": "choice",
"choice": "right",
"confidence": 0.68,
"probabilities": {
"left": 0.16,
"right": 0.84
}
},
"pair_8_11": {
"type": "choice",
"choice": "right",
"confidence": 0.4,
"probabilities": {
"left": 0.3,
"right": 0.7
}
},
"pair_8_12": {
"type": "choice",
"choice": "right",
"confidence": 0.84,
"probabilities": {
"left": 0.08,
"right": 0.92
}
},
"pair_8_13": {
"type": "choice",
"choice": "right",
"confidence": 0.6,
"probabilities": {
"left": 0.2,
"right": 0.8
}
},
"pair_8_14": {
"type": "choice",
"choice": "right",
"confidence": 0.72,
"probabilities": {
"left": 0.14,
"right": 0.86
}
},
"pair_8_15": {
"type": "choice",
"choice": "right",
"confidence": 0.98,
"probabilities": {
"left": 0.01,
"right": 0.99
}
},
"pair_8_16": {
"type": "choice",
"choice": "right",
"confidence": 0.88,
"probabilities": {
"left": 0.06,
"right": 0.94
}
},
"pair_8_17": {
"type": "choice",
"choice": "right",
"confidence": 0.8,
"probabilities": {
"left": 0.1,
"right": 0.9
}
},
"pair_8_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_8_19": {
"type": "choice",
"choice": "right",
"confidence": 0.9,
"probabilities": {
"left": 0.05,
"right": 0.95
}
},
"pair_8_20": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_9_10": {
"type": "choice",
"choice": "left",
"confidence": 0.98,
"probabilities": {
"left": 0.99,
"right": 0.01
}
},
"pair_9_11": {
"type": "choice",
"choice": "left",
"confidence": 0.98,
"probabilities": {
"left": 0.99,
"right": 0.01
}
},
"pair_9_12": {
"type": "choice",
"choice": "left",
"confidence": 0.6,
"probabilities": {
"left": 0.8,
"right": 0.2
}
},
"pair_9_13": {
"type": "choice",
"choice": "left",
"confidence": 0.96,
"probabilities": {
"left": 0.98,
"right": 0.02
}
},
"pair_9_14": {
"type": "choice",
"choice": "left",
"confidence": 0.98,
"probabilities": {
"left": 0.99,
"right": 0.01
}
},
"pair_9_15": {
"type": "choice",
"choice": "left",
"confidence": 0.56,
"probabilities": {
"left": 0.78,
"right": 0.22
}
},
"pair_9_16": {
"type": "choice",
"choice": "left",
"confidence": 0.88,
"probabilities": {
"left": 0.94,
"right": 0.06
}
},
"pair_9_17": {
"type": "choice",
"choice": "left",
"confidence": 0.98,
"probabilities": {
"left": 0.99,
"right": 0.01
}
},
"pair_9_18": {
"type": "choice",
"choice": "right",
"confidence": 0.72,
"probabilities": {
"left": 0.14,
"right": 0.86
}
},
"pair_9_19": {
"type": "choice",
"choice": "left",
"confidence": 0.68,
"probabilities": {
"left": 0.84,
"right": 0.16
}
},
"pair_9_20": {
"type": "choice",
"choice": "left",
"confidence": 0.62,
"probabilities": {
"left": 0.81,
"right": 0.19
}
},
"pair_10_11": {
"type": "choice",
"choice": "right",
"confidence": 0.16,
"probabilities": {
"left": 0.42,
"right": 0.58
}
},
"pair_10_12": {
"type": "choice",
"choice": "right",
"confidence": 0.76,
"probabilities": {
"left": 0.12,
"right": 0.88
}
},
"pair_10_13": {
"type": "choice",
"choice": "right",
"confidence": 0.46,
"probabilities": {
"left": 0.27,
"right": 0.73
}
},
"pair_10_14": {
"type": "choice",
"choice": "right",
"confidence": 0.66,
"probabilities": {
"left": 0.17,
"right": 0.83
}
},
"pair_10_15": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_10_16": {
"type": "choice",
"choice": "right",
"confidence": 0.86,
"probabilities": {
"left": 0.07,
"right": 0.93
}
},
"pair_10_17": {
"type": "choice",
"choice": "right",
"confidence": 0.66,
"probabilities": {
"left": 0.17,
"right": 0.83
}
},
"pair_10_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_10_19": {
"type": "choice",
"choice": "right",
"confidence": 0.92,
"probabilities": {
"left": 0.04,
"right": 0.96
}
},
"pair_10_20": {
"type": "choice",
"choice": "right",
"confidence": 0.98,
"probabilities": {
"left": 0.01,
"right": 0.99
}
},
"pair_11_12": {
"type": "choice",
"choice": "right",
"confidence": 0.86,
"probabilities": {
"left": 0.07,
"right": 0.93
}
},
"pair_11_13": {
"type": "choice",
"choice": "right",
"confidence": 0.48,
"probabilities": {
"left": 0.26,
"right": 0.74
}
},
"pair_11_14": {
"type": "choice",
"choice": "right",
"confidence": 0.6,
"probabilities": {
"left": 0.2,
"right": 0.8
}
},
"pair_11_15": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_11_16": {
"type": "choice",
"choice": "right",
"confidence": 0.88,
"probabilities": {
"left": 0.06,
"right": 0.94
}
},
"pair_11_17": {
"type": "choice",
"choice": "right",
"confidence": 0.64,
"probabilities": {
"left": 0.18,
"right": 0.82
}
},
"pair_11_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_11_19": {
"type": "choice",
"choice": "right",
"confidence": 0.94,
"probabilities": {
"left": 0.03,
"right": 0.97
}
},
"pair_11_20": {
"type": "choice",
"choice": "right",
"confidence": 0.98,
"probabilities": {
"left": 0.01,
"right": 0.99
}
},
"pair_12_13": {
"type": "choice",
"choice": "left",
"confidence": 0.64,
"probabilities": {
"left": 0.82,
"right": 0.18
}
},
"pair_12_14": {
"type": "choice",
"choice": "left",
"confidence": 0.66,
"probabilities": {
"left": 0.83,
"right": 0.17
}
},
"pair_12_15": {
"type": "choice",
"choice": "right",
"confidence": 0.06,
"probabilities": {
"left": 0.47,
"right": 0.53
}
},
"pair_12_16": {
"type": "choice",
"choice": "left",
"confidence": 0.08,
"probabilities": {
"left": 0.54,
"right": 0.46
}
},
"pair_12_17": {
"type": "choice",
"choice": "left",
"confidence": 0.32,
"probabilities": {
"left": 0.66,
"right": 0.34
}
},
"pair_12_18": {
"type": "choice",
"choice": "right",
"confidence": 0.84,
"probabilities": {
"left": 0.08,
"right": 0.92
}
},
"pair_12_19": {
"type": "choice",
"choice": "right",
"confidence": 0.26,
"probabilities": {
"left": 0.37,
"right": 0.63
}
},
"pair_12_20": {
"type": "choice",
"choice": "right",
"confidence": 0.48,
"probabilities": {
"left": 0.26,
"right": 0.74
}
},
"pair_13_14": {
"type": "choice",
"choice": "right",
"confidence": 0.16,
"probabilities": {
"left": 0.42,
"right": 0.58
}
},
"pair_13_15": {
"type": "choice",
"choice": "right",
"confidence": 0.9,
"probabilities": {
"left": 0.05,
"right": 0.95
}
},
"pair_13_16": {
"type": "choice",
"choice": "right",
"confidence": 0.58,
"probabilities": {
"left": 0.21,
"right": 0.79
}
},
"pair_13_17": {
"type": "choice",
"choice": "right",
"confidence": 0.36,
"probabilities": {
"left": 0.32,
"right": 0.68
}
},
"pair_13_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_13_19": {
"type": "choice",
"choice": "right",
"confidence": 0.52,
"probabilities": {
"left": 0.24,
"right": 0.76
}
},
"pair_13_20": {
"type": "choice",
"choice": "right",
"confidence": 0.92,
"probabilities": {
"left": 0.04,
"right": 0.96
}
},
"pair_14_15": {
"type": "choice",
"choice": "right",
"confidence": 0.86,
"probabilities": {
"left": 0.07,
"right": 0.93
}
},
"pair_14_16": {
"type": "choice",
"choice": "right",
"confidence": 0.68,
"probabilities": {
"left": 0.16,
"right": 0.84
}
},
"pair_14_17": {
"type": "choice",
"choice": "right",
"confidence": 0.14,
"probabilities": {
"left": 0.43,
"right": 0.57
}
},
"pair_14_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_14_19": {
"type": "choice",
"choice": "right",
"confidence": 0.8,
"probabilities": {
"left": 0.1,
"right": 0.9
}
},
"pair_14_20": {
"type": "choice",
"choice": "right",
"confidence": 0.94,
"probabilities": {
"left": 0.03,
"right": 0.97
}
},
"pair_15_16": {
"type": "choice",
"choice": "left",
"confidence": 0.16,
"probabilities": {
"left": 0.58,
"right": 0.42
}
},
"pair_15_17": {
"type": "choice",
"choice": "left",
"confidence": 0.38,
"probabilities": {
"left": 0.69,
"right": 0.31
}
},
"pair_15_18": {
"type": "choice",
"choice": "right",
"confidence": 0.88,
"probabilities": {
"left": 0.06,
"right": 0.94
}
},
"pair_15_19": {
"type": "choice",
"choice": "right",
"confidence": 0.04,
"probabilities": {
"left": 0.48,
"right": 0.52
}
},
"pair_15_20": {
"type": "choice",
"choice": "right",
"confidence": 0.08,
"probabilities": {
"left": 0.46,
"right": 0.54
}
},
"pair_16_17": {
"type": "choice",
"choice": "left",
"confidence": 0.28,
"probabilities": {
"left": 0.64,
"right": 0.36
}
},
"pair_16_18": {
"type": "choice",
"choice": "right",
"confidence": 0.96,
"probabilities": {
"left": 0.02,
"right": 0.98
}
},
"pair_16_19": {
"type": "choice",
"choice": "right",
"confidence": 0.46,
"probabilities": {
"left": 0.27,
"right": 0.73
}
},
"pair_16_20": {
"type": "choice",
"choice": "right",
"confidence": 0.66,
"probabilities": {
"left": 0.17,
"right": 0.83
}
},
"pair_17_18": {
"type": "choice",
"choice": "right",
"confidence": 1.0,
"probabilities": {
"left": 0.0,
"right": 1.0
}
},
"pair_17_19": {
"type": "choice",
"choice": "right",
"confidence": 0.52,
"probabilities": {
"left": 0.24,
"right": 0.76
}
},
"pair_17_20": {
"type": "choice",
"choice": "right",
"confidence": 0.94,
"probabilities": {
"left": 0.03,
"right": 0.97
}
},
"pair_18_19": {
"type": "choice",
"choice": "left",
"confidence": 0.86,
"probabilities": {
"left": 0.93,
"right": 0.07
}
},
"pair_18_20": {
"type": "choice",
"choice": "left",
"confidence": 0.84,
"probabilities": {
"left": 0.92,
"right": 0.08
}
},
"pair_19_20": {
"type": "choice",
"choice": "right",
"confidence": 0.5,
"probabilities": {
"left": 0.25,
"right": 0.75
}
}
},
"usage": {
"input_tokens": 15646,
"output_tokens": 6292
}
}