marola-cli
API
Generated with
marola-cli/marola/marola.bench/OceanBenchmark

OceanBenchmark

marola.bench.OceanBenchmark
object OceanBenchmark

Is marola's answer better than just asking the model?.

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any
Self type

Members list

Type members

Classlikes

final case class ArmSummary(arm: String, coverageInCorpus: Double, coverageGeneral: Double, coverageAll: Double, citedPct: Double, abstainedPct: Double, meanMs: Long)

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
final case class Question(id: String, topic: String, question: String, keywords: List[String], inCorpus: Boolean)

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
final case class Report(model: String, results: List[Result], summaries: List[ArmSummary], markdown: String)

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
final case class Result(arm: String, q: Question, answer: String, coverage: Double, cited: Boolean, abstained: Boolean, ms: Long)

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Value members

Concrete methods

def coverage(answer: String, keywords: List[String]): Double

Pure: fraction of keyword groups (a|b = any of) found, case-insensitive, accents ignored.

Pure: fraction of keyword groups (a|b = any of) found, case-insensitive, accents ignored.

Attributes

def isAbstained(answer: String): Boolean
def isCited(answer: String): Boolean
def load(): List[Question]
def render(results: List[Result], summaries: List[ArmSummary]): String
def run(store: KnowledgeStore, llm: LlmClient, minScore: Double): Report < Sync
def save(report: Report): String
def summarise(arm: String, rs: List[Result]): ArmSummary
def verdict(summaries: List[ArmSummary], results: List[Result]): String

Deterministic verdict + "how to beat it" from the numbers — no model in the loop.

Deterministic verdict + "how to beat it" from the numbers — no model in the loop.

Attributes

Concrete fields

val Arms: List[String]
Generated with