Skip to content

Instantly share code, notes, and snippets.

@johnfelipe

johnfelipe/log15082027.sh

Created August 15, 2024 10:07
Show Gist options
  • Save johnfelipe/62fa83fbab496da698967587a6e2b774 to your computer and use it in GitHub Desktop.
Save johnfelipe/62fa83fbab496da698967587a6e2b774 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
log15082027.sh
(livebench_env) root@template:~/LiveBench/livebench# python gen_ground_truth_judgment.py --bench-name live_bench --model claude-3-5-sonnet-20240620
models: dict_keys(['anthropic', 'claude-3-5-sonnet-20240620'])
Stats:
{
"bench_name": "live_bench/coding/coding_completion",
"model_list": [
"claude-3-5-sonnet-20240620"
],
"total_num_questions": 50,
"total_num_matches": 50,
"output_path": "data/live_bench/coding/coding_completion/model_judgment/ground_truth_judgment.jsonl"
}
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.48it/s]
question: a902525e385453571494ab18eaac931c67b171c8106286ba55221bbc89aa1d3b, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:04<00:00, 4.54s/it]
question: 049ed4055f772aa9bccfda47ae425ee8a3c16d2c74330818d0397890795ea7fb, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.24it/s]
question: b122f7e6e1fa155665d3553c4cddf946c021fe467225f39dbd4dec56b813db1d, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.35it/s]
question: 298828230337781810c328f815976e74c6c89c7b14b22beea1cb3d0dc1a7bcae, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:10<00:00, 10.77s/it]
question: 59c63e484f7d21595c2122342662bfc1b0c57f63f168f96fc45a27dc7bbc0a07, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 11.05it/s]
question: 009f4f7e8275c73de2eae601d76ff7163f613880788d870c1484b1c2a1383fbd, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 11.94it/s]
question: ab88e681245c5c93976393e5ee7f61a122aa12f9f5d970701d874335bf5a6362, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:01<00:00, 1.66s/it]
question: 16503edeb04a3a1af42e446248632951a149ca0cb3253c2e4fc91716ad53b5c7, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.58it/s]
question: 9e895f817a6a867671d8735578f058a34a19dde837957b1a9c9347d7e13a8e38, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 10.08it/s]
question: ebde77f9af990a37740871b38788987a907d4ab08556978324a42c3293bb68a9, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 11.23it/s]
question: 7ae9d085e0923cd5063163ca2628c4b9ecdf0a7ca192f3806bbaf16c735d4946, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.21it/s]
question: 59811e99700ace9d7a98d352e7970f6cd244e25344d349c12e96b18067777d54, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.89it/s]
question: 6f128ce8389efae085f84f835f4678a214c45ca94cf7933178b07562789bc4d7, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 11.14it/s]
question: e25750fd4c8fae19856e77a875093ab90a6333a680ccbd45dbabdaddb205bf2a, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.60it/s]
question: e1553e31f9f0e93848a8e06d452fa0d6b8c4bf21580def1d98237dbac399baca, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.80it/s]
question: fe6312c9e8cb798517e3695c245cbbfbf4fd6dc737903628abc4a2b55c3f4af5, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.37it/s]
question: 786432ddcf555d879432b906560955b9d464831f713946ce0210dcba478becd2, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.57it/s]
question: 37a9bdb633709429b87eae7e703feaf3ea5a8bd9ea41a33cb23c4c657d918768, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.80it/s]
question: 9ccb5f2d887b739bf9fa290e9b3098ed5c86da0de11b50b8128de0f8a7ff8e42, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.46it/s]
question: eae492105162057ac07f4920749d2125f86b5a335557c61d328d49f198af3343, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.88it/s]
question: 541fd30fbc3ff5b8845fdef7e228778a903adab9d122280003aa2e8596aa2c63, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.56it/s]
question: 4f4e5b4d83cbc334fc40c3702b6abaaaa5c87ddf73d3103a8e0ef6c8002ecb01, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 8.97it/s]
question: 4b6062ff2bb0859fc70acfbdca5f67617a5d5e631cb2fc650b8f641c237acb44, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.90it/s]
question: a62c56a25fd1a6fb193268c07d5fa714d8fe8ef25aa8884d4f910e3f58dd51fe, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.35it/s]
question: 15b97089ff238df02cdec621f3bd0d14e0b096e4d26d463e2bcb66a0fcb57a9c, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.24it/s]
question: 945b77d39f64bc7e9076de58b48806c99c26ca7a45f94737fa80f0702625969c, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.19it/s]
question: 7795fc4543dfb60b56345359d525454c6ec3af02be98fe13f98b34af41356500, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.02it/s]
question: 030471ea2fc50a29f82560555033ceaec6b127c9ad0f68d34565ba221279d54b, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.90it/s]
question: 5ced71fafeb7058ec2edec54fe4022058381689584152f36fba4aec109081931, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 11.03it/s]
question: 26aa1d9bdee3ca21ecb30b18871b017f5013da90f96af113e433e9172f600342, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 12.11it/s]
question: 368b91cf69b159b1a3ef4740009cb201b46ea95478c9c5fc17df884b825ccacb, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.75it/s]
question: 7878539e8dd4fe1d4bc66a440e16c102d66f158fed7828a84f3765505ba8f1d3, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.79it/s]
question: 3cd8c16a338ab1b2d081add09e47c05cd4bad6c4ddd20c12356d911502fc54a0, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 7.89it/s]
question: e2ee8ea63cf9d79d7695c7e310de966d5cf7f96f103b706faa930557991fc816, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.87it/s]
question: 45d51d0927f37783772c554d860a9c17db83a3e8e31dd02abeb93150acdaba55, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.72it/s]
question: 447b358040b99c91a12a17ec5e81ee3bdc257155200b3cec64c115166f7e12a9, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.62it/s]
question: 73642dac6679890d1c7303e79ee6dbc93b28c3943fd8733371bcf9e73cb06948, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.00it/s]
question: 100c6ce0a28db0672fc9a23207f72ad7decfc90c81f03baf2ecdbb30297993a3, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 7.46it/s]
question: 1693afc152bb5f955ac7298d7381160294b7a32c0844c02817d7a66a0e028750, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 10.62it/s]
question: 2ac5a0878eb181cefc6922338f7b043368fa897d089955b97eb0b6210f94f3e9, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.45it/s]
question: 6755f738cebb3ae683e9dcd2e56007bfac992a804648cc315cd88d8dc13a17e7, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.24it/s]
question: 317b199a6aeaec75555a76bac287b3f795cc968930677de0fea1d039b055e90e, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 10.13it/s]
question: 795dd501310981efaa1cfe050e20d249d845dbc90df7e7c9b07bd1b1925b5263, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 10.72it/s]
question: 4952a5d420fa195e7b041251e85bce64c87b95e8f1cec05d7e16ce149e88c967, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 9.52it/s]
question: 5c9d7ade302709f4bdcf6e3b64127d1e67a3a3078b3a03ef241329547cf320c6, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 10.88it/s]
question: 5f7ac157843831b50cc45eb8991c1659e0196805ea75d99202624fa06cd61342, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 10.51it/s]
question: 7ad613404830cbc9bd478f52fb74f5c73a2c55405938406316a2409e744783c8, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 8.55it/s]
question: d5ecf69bb5d8e62278ed3d129812cdc0e9bfb90d8b6ca369847e6ed19edffffb, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 11.23it/s]
question: 0119bdbf9e79f05e988b6d3408e2fdc0b7e4bc11e81ff9f24bed96268c503801, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 10.47it/s]
question: b8e56097d9b5740294ef468314833c875b1ffe610edae1c1c79b91f7c09b5ece, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|███████████████████████████████████████████| 50/50 [00:27<00:00, 1.83it/s]
models: dict_keys(['claude-3-5-sonnet-20240620'])
Stats:
{
"bench_name": "live_bench/coding/LCB_generation",
"model_list": [
"claude-3-5-sonnet-20240620"
],
"total_num_questions": 78,
"total_num_matches": 78,
"output_path": "data/live_bench/coding/LCB_generation/model_judgment/ground_truth_judgment.jsonl"
}
100%|█████████████████████████████████████████████| 1/1 [00:01<00:00, 1.02s/it]
question: 2cb17950d950274fe6858fb7157bc44e160006e47657e9ca7d3b669ec9a0ff07, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.49it/s]
question: 5d2c5a95687741962e9fa5f3a5c8ca8cda81c66617f61c3ce119349914f9f1d4, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.42it/s]
question: 7f4101a7131ddc370238e7180e5a6b07a534141417b01a5b83a9ab33ea26b0d6, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.33it/s]
question: f78df88e6755702752e1c216f9734cbf1803a006546503128acb8747aee1a738, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.87it/s]
question: 556eb10bbb584e5dc57d787eede0ca725d3d509ff39849a89b568e4ca9f2dc8e, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:05<00:00, 5.61s/it]
question: 588bc5f2d796806fc8d50bb1746f5bdf495dce0f1a752ea2dc5fe89d3e1b5f60, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.84it/s]
question: d9c8393373b3a80ed8f75ad3fbfbe74740f1799b3f2ae7a12486ec41a14aca53, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.23it/s]
question: df2028f3f71248ce456f02f00f5e542660904738e6cd45061a0c7c5662c3d2b4, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:02<00:00, 2.65s/it]
question: cf932b62f91764e164fa14f8b4f50ad8346b1b653765fb9fdec13d372b10be73, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.85it/s]
question: 640e8e5a789f35ff19127fc822b980499f8db1eb0127e0ad642f0f01c425e463, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.78it/s]
question: 6a84becb148316106cde53436e3c0b4033d7957dc1e2e3e2d7d30e712a49ecbf, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:02<00:00, 2.25s/it]
question: 30cd4633ff2a6f4acc868ddd9dcc076c4a8eae5a7a18c736c9c7504c1e55a516, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.89it/s]
question: b6a1eeb2bf7e0899432085c68d3f2a5e2395cb6adf1334495db344b742950c6c, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.95it/s]
question: 2a0f52cd0d54e167a9d1b9bcb7fcb76d565f37019a19971b7377297c5835cec1, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.96it/s]
question: 67a7d1cc72639935a09a3256d5a1aec2e55403972912391e7f5f079e69ee1cc7, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.86it/s]
question: 03741bcada4e970eedeb7f28fdf45c496779b98f3df8e513f992a51cf072f67c, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.58it/s]
question: ffdb71c563e5f8e936ab2649a069bfbdbebcde991c6ae11a2e83ec18c0d8e50d, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.88it/s]
question: 306525ba191d9f60529c2b5a3b3fb659106ca56db058b66b4181904705a84aab, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:06<00:00, 6.90s/it]
question: 3d2aec36cd900b962211c24ae69daf2692f7200fe06d34c7a236d531d0a3c17f, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.27it/s]
question: ccf40d3ba5eed106abc518f7c58a4a72b5c1f732ec33cad5e0617b54ff46fa22, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.53it/s]
question: d5213885ecf30ec0d5b6fa5e53af31eb5ed207d3257b2ae8843b1eed735ac7a5, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.91it/s]
question: df00a4688e96b001f32ffd39913ae151bfa14034bdca636193c96974529f7cf9, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.68it/s]
question: c675d9db6b456f793da486f16102468ed1e241bbb7846039608b2c0f9c5d1012, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.70it/s]
question: 5a87f768962bad5e67430639470f03835d0f2abf3fb39f38cf96145ce768f485, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.11it/s]
question: 3375d42f6db384f62b159eb89a85d715736790fd1abb380760ffbd12339fd4d5, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.18it/s]
question: e4ee35e76ddb5c90ccea3e6aa41f00de01b44122d4d602dd4a0da6892757e3af, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.04it/s]
question: db5240d8bb9da989f3ffea465538592857f8fa5b7d4293579356c8456253e721, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.55it/s]
question: 5afb74e155e6538cd76e64fc9df21bd78073b280f0b95b675cbb86aa877b2a59, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.60it/s]
question: 55893da1c59947e4e7469ad4b0683142b475556a6368135400c20a3d8f43def8, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.07it/s]
question: 7439c76752648b2b25924c0810d91f130b7bea2192c80a2603d5662b0685391a, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.07it/s]
question: 89ca002eeeb0c60dbdd4ba3c8129b1d38672338a0375643ac3f942eca79bc7cc, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:06<00:00, 6.99s/it]
question: d191855119f72c53a37db900f0fbf8d177bb12c255da8284af51335e57272f83, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.49it/s]
question: d761bf4d5741e128fca776f16db3dd1d6265291dc5dbe0831033d5918188ada0, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.15it/s]
question: 042b958543e9a61fd083dfd6067a91845b8e272295a282512ec9b527cff2e113, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.70it/s]
question: 388e39eefc4d3950ad99966e9c8dfbf64b59991d7e6cc2d9fb0a617f4025475e, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.88it/s]
question: e4386373494a3b0a70dc5acb7d3a21eb4dc8dc53d4afafcc2d057d16fb68a0a1, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.18it/s]
question: 232d3a9537df2b3af811ae7ba1baac43ebf6a1971c98c7bf0ea84d73eba4c506, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.16it/s]
question: 39d2dcf38a69f076f8cd3af146650328044014addb6c2f6abd7ae51d56c56869, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.63it/s]
question: b8637ee53e144cd0f68a891e7962df2b44c1949cbbdf46670c49b92f29b8bf46, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:01<00:00, 1.61s/it]
question: af9c90d1566582311697d76693eb2be23401bc8fbb7e8b0e595a4080a53e3fa6, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.87it/s]
question: cd2f643fc3de0dbcb738ea9f8ca03f738a76bea2110d7dd97ccf0d8e418a8694, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.64it/s]
question: 87871f3ecdda8c4583b2834741bb312a3917c9035cd21d151902b9eae123e17c, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.92it/s]
question: 9026b23d1094bb0072bae31bd66f9a8cc7220e3951dc520c53a2f21c4e8a472c, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.19it/s]
question: eecef5ebcd4b0224ec2404b15759eae2703c32db82dc7fd711af23c0dffc888a, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.80it/s]
question: 0664f46bd7bf6217d229be6762834af2aa642e7b97b14469f478c93a382ee520, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.83it/s]
question: f62daa0a810a70c8048484566eb0dcfb331b5299ce2a104f71cbbed4a98ed2b1, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.26it/s]
question: 65369f81842e0735d0dd6619a120d5760217c9b8ac481de89cc20983ea555953, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.58it/s]
question: 6cb0c2d9892875f79d632e273efd7ae3fa4ccb324fbce753673b8314aa80c737, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.04it/s]
question: 88d41d97990a9e7f990eb8eb71440d007fd194758ce2c3fe00b77e186fd65c3d, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.99it/s]
question: 507f6dbb4e7d52e6f645b33aa73096e076769e196010810eeca84820375d4211, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.71it/s]
question: 7ac5182f5b8a86a644e912d6146da1290f785382a6c178512d8348a055e0a976, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.58it/s]
question: 412707cd401e505990ba7dcc4fae0938537212b798cafd0cabd0ec75220baecf, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:01<00:00, 1.69s/it]
question: 75a3057122fddbd336b3e9a664d6254556cc1854312b07188cb719b9636a0acc, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.15it/s]
question: 5a17ebccc62403424aaaa96b3fe4b93604c49ef84c15556490d7e86b10b5126f, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.70it/s]
question: 63a5960e3858d9b8ca8f734d21ac4828a9d96f67596da0b29f7b3c6b1298aee2, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.01it/s]
question: 0c1688303f94019295a1fe3d5d691494e6226647d863fbaf7205cd12573d1dd5, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.00it/s]
question: 3e0ba617402710c701decb2edec4b14fb15a84d74dd02510fd932f3902be1827, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.50it/s]
question: 832eb791ff0dae774b2f1d75ba01356d9f49fead0dacaa466d83fb1e59a8dd78, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.35it/s]
question: 09c0c67ed82f00989b1c9c48646e2c31c89b81dd2afc922cf241ba082afef18b, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.78it/s]
question: a762c0ca6500a33a975869105641d3d4fbaf897051bacc2d6fbe83bc3de23bd9, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:06<00:00, 6.44s/it]
question: 3de555bbcef9919a92a742dd969301eaea38baee624b4c4851e6a495cc83b3cf, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 2.19it/s]
question: 109469ec1296f25ee25bc3a09e4265a5df823816b438d97cb3d2ba15717626b5, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.69it/s]
question: 89cad09d866391980ddbdbe8208870d1792916fc0e7862f2f27744153ccf9eea, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.50it/s]
question: 8b720a5b61bea0e6a0dcfba224824fff2f48c1d21ef3ae9695d3ec9c84f621d1, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:06<00:00, 6.59s/it]
question: e1e226d29b9ab59fc2de2d719a0e7117572f7bd9dc375709a92a673987c1b29d, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.23it/s]
question: 5a126374e3885ec262a06ef4f7eaafd34477f3bc68ba8e6b04123c2c32e7352d, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.82it/s]
question: d414f7768eb066f524236800aa2dbb8a4554f2e4abc22a99ef88fe30cbcc308e, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.79it/s]
question: 6afe4974569ffea0733fb153fa9bc6ea75c6ada73e3095b7a3efb65fabbfae06, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
87%|█████████████████████████████████████▍ | 68/78 [01:04<00:10, 1.07s/italarm went off | 0/1 [00:00<?, ?it/s]
100%|█████████████████████████████████████████████| 1/1 [00:06<00:00, 6.18s/it]
question: ef2c6b4ab2f09b2f310eff0a1e7c7e22e9ed12a554aa667dc9d7162fdb123f50, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.65it/s]
question: 01c73e7f5bd7b486915f97829994aca9b95d6116c510d9f3781b5399dea066bd, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.62it/s]
question: 23132a8725620697e72843b28d711947911252925ddaec701e17d85564165c4f, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 4.25it/s]
question: 27c31c63ff778ef9594589756be22b75959369914a5b4fae17fa3255927e1dc4, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.14it/s]
question: 89101e97bd95e50c1d5d9db784d73f2894a20e4c2ed3e67cf124c7959f8ab4dd, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.13it/s]
question: 176daded29d196e2697bff1351ef4a77c31ffad3757b20405a6c7a346af31ddb, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.51it/s]
question: a326807520b5d18b9a6fa534c086ba3ba53ec6481d9a30222b1b09a6b6699483, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.73it/s]
question: e882b58cb776d5ecbd45f20da44a2628df326ee12c927f4ebb751222629601b1, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 6.43it/s]
question: 7c1f49314f2c261404fdd7c795620c2fc92d4545bdccc3fc59a15cf3193dba0b, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████████| 1/1 [00:06<00:00, 6.53s/it]
question: 992a18bff05e5db69cfeadf0d02b6dfd2808c0997c86e3cc1abb5a4fbd550c70, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|███████████████████████████████████████████| 78/78 [01:19<00:00, 1.02s/it]
models: dict_keys(['claude-3-5-sonnet-20240620'])
Stats:
{
"bench_name": "live_bench/data_analysis/tablereformat",
"model_list": [
"claude-3-5-sonnet-20240620"
],
"total_num_questions": 50,
"total_num_matches": 50,
"output_path": "data/live_bench/data_analysis/tablereformat/model_judgment/ground_truth_judgment.jsonl"
}
0%| | 0/50 [00:00<?, ?it/s]question: e114ea800daa6938bd7bbc29c6fde32844324662764b5cb63d7e4e78c3b66c65, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 09752b3d3e355017282301de1735bd903221368e1fadf3e64aa9594ef7730e62, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: eb8aebddc3e1eff35a92de9e8306dfcfebd25201eefda2921b830226b5347dc5, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 57772b38ab8d4f9f2e30d5e0cca6e007f228786523ed027bdbf37b59fe20e3b1, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 8b25c07592f100c7c61af45d3de6eec07f4996d0a211f613cc1e224db02bba4c, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: ce1fa8f673c3d33b3986e4f4f148ab1a11e42cd2b7f15390f0f4f2acb530a6e3, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 56ace477670fa1527771dc1a4f2babac3b704f1c313b8981a53ce892f55b6c05, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 3f9091ebb24ea69d1e9ad7a20e3a617f47548595e9b0b0a46a95061ec3e81740, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
16%|███████ | 8/50 [00:00<00:00, 73.91it/s]question: 35b3098f43e9129b80af5263066ffad973670da10a3bbc4d350fa88cea4d980f, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 0cbc79763d1930cd7c78821f52cbb8c368eacbee9ea3b9bb9ece1e79167deb4a, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 2379e1e2586eacdf0c9ea0b9385f3c679ff72aa7e2a74fb741aa42018b4d78ea, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 23369207a1755bd3b7cea52155e7cbbd7ab1e2fa79015d1f8776111b8648f8ed, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 9dd93740e50a0913119103c1212284600703756ff930d2a8fa46d3dc97912d96, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 8f3ca4d439a2167eda91a41deaecf48838405ce32967097d2ec7e931b1313cf4, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 31b5500fbd88c9b6087f15229a84578b6863700ef5b4bf2c645d8927a4723a77, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 9b8b3c1cdfbadc5d248114dcd74f7376fe3e24db964f3acb8d12159404199aac, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
32%|█████████████▊ | 16/50 [00:00<00:00, 68.92it/s]question: 1f592e4e1a40499fb15905f6badb7c507a643106aec8d907f34de9cd200cb3fa, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: e852443f6993386ec44106f68bee0f7f278cfd9fb116228e55a50713257692b2, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: f44bcc507aa7a438c07f435c70e687868c07af785cc257410780ff861c54c646, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 0bedfad80bcaab18b0ab15531247a61a8b75f42c6e87c40f05d398dc25984d35, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 9cd37119651a821e2695ee073ddf004d50d9add830f4e7f3bc469f9b0d4ddbe3, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: af06a250c4d58799cd7bf0f73df94134106098b21d8b0a3a3e61dd9eacda6724, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: fe2193c57ea45001a4926fe79284b5a1405531d70f8ff5e5a4ebfeea8a79a10e, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: f2dcd6a353c4390c3d98c8d4ff03d778f00d0d6c6b9f8238af4f09f81f6d9924, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 4d22f4f91dfc8188c2244048d968e9885ee063658c14fcf43c8156983f5a395f, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: e321ff8846ddd951d029b63efd6f0cdadb5b6daae266e3b21a6f3e805faf75e4, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: dbc100e6b17f59547b155762e53564c1d30e21197f86780b166e4c067ee4b0e8, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 083282355242eb434e4c4559eabea700f94928fd2e1d0d4df6a59ee143866e60, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 804182061bd3648a5d1079e9836aa8cb7e9201a32f190863551299075fbeac47, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 16b99f3754472adfba51046b35d0bb8af8d7e95e8dbac50d5f0f82b9775229df, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 00d7be878c842d12814cb113caf8503525f8cf845b7d3ca4b8387c843f06ebc9, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 0e5a359e5a70462b866ffd40a10f0b976ed63030be062c8a9eb8c72316ac2146, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 0d102a277bdaf52b40a8dc150408f122828ee63f2a55dd0e58f3c7b51933d345, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
66%|███████████████████████████▋ | 33/50 [00:00<00:00, 109.16it/s]question: 0a9d3c9d94cdbd52adf5852ebdf291b4ff8788032d950f62695d1dcc298b54f6, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 07844eb9fb31c8e9cac12e29662d01c221f762a67418b020ff4eae637065539a, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: c42ef3e7297ebddb097e9aa4d5527dce29367f617c29bf7144de2633107ead00, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: bfe58cf09204ef9dddeb7358323fbab09c078fbc88d022c2387e0eada4470849, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: b64b2155d2e4e74cbb52dcb6f97298cbf28eef0159973600aecdc80a6c49c8df, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 132dabf6ac92193bcc1b1cab0080ee5531ab45a959eefc1e5b3cbf57976bcf9a, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 40af2768cabe32744e3b1efd2552edb077e2539a8f45808852e7d83147a82519, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: ed3a1cf09c7eefe66d4775384c633a1bbc48f09a7d2257028362479f057d7e3e, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 0a03666dc72cf14a5cf569f517483315380edb796ac0394c3c204ced2e4a7428, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 6a71a2fd132bae45c6ef2fc93d0bfcf3d4f71025db07ec88fed08ff83b4eca45, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: c55626b8561ca43f0fdf11e24b2083c4d3180a83c87ea6171a159a1e82f1da54, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: ed588c8250de9be2b31be0bc4c7820f3fa97c6084f822bce03d324f20d4c228d, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
90%|█████████████████████████████████████▊ | 45/50 [00:00<00:00, 106.08it/s]question: 6204f00c8a38e299225434e2218dc1fbf65eb3040ed083a97c048e4f3b7dc2c8, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 4810de734ddf549ae44d69fe3717e2ad95593f88a0f7d72211f46cbdd22ad513, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 77765399a07884782f5a539ccb9e8820f5c15a090a666f59b21f804706ecadc9, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: b78c258e2cfd2954eb6ff290f39427d6270c69d57cc36e8a1c31839de39c885a, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 26fbcb603a637ccd27d65387509e548e532334c5895fd56d7a9d531cc1b125fb, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
100%|██████████████████████████████████████████| 50/50 [00:00<00:00, 104.51it/s]
models: dict_keys(['claude-3-5-sonnet-20240620'])
Stats:
{
"bench_name": "live_bench/data_analysis/cta",
"model_list": [
"claude-3-5-sonnet-20240620"
],
"total_num_questions": 50,
"total_num_matches": 50,
"output_path": "data/live_bench/data_analysis/cta/model_judgment/ground_truth_judgment.jsonl"
}
0%| | 0/50 [00:00<?, ?it/s]question: d4ec8efff8fdcc6db682bb2c9dc2b5284ea7ca5d0f79663832e203e3d52bd125, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: a0ef4e780ad34fa8a80b2ce6367a36c65899cfeb5e610e896857e49bc240e45e, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 48dd183d63a78a751541e8d237cfbfaeeba2df8cd7f0d6fe58324d74aad9ff3b, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 567d5f634453da734fb7ceab3bbea4dd283ac19a125102fe9b533ca5e0e388e5, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 5c3dfa6b8c0ecd07ea0091b21fb237ade69bdce3c3a9cdeed307bee1e968ce2b, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 051ed5edf44bb798385076a1260de95b272dd2e0f5167dc78e514ce434af3ef6, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 811f288b7c362542153770a32060519cf59d30d4c61368bb9abea3a56f873a09, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 3f7b12f0c920812c39c3217750021c9e9153c1934d4673e5aaf481be74f89aa9, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 13ca9b1d4d1937587bd2cc18ac8804a4c57d5e9066a6d7501d2f06ab33119cb6, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: f9bcb466e175b91a55ff30f1265ad410a93737012e3e6c8f288adcd3525f5d7e, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 2cbf1cc153f0500650d2b4ce15643bc7319268df28b56624014250dc8fe28b07, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: f6239269d04fcfd9a96e904b2016c00b132487ce74c331d77b829d5cdb6f5df7, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 60d902d2b18d8ed747f135fd78d13c2da523d6067ed3865723d5e34a99abdf61, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 76b0d923c1e41da8c302c906ce9b145c4f648e04442ee224d899de34e2a09c27, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: b75684697587c1ce05a1377916ae8da11e579e9e8a3d9e693a955a1dc8522f2e, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 666f26a703b286a7d31f2f46070307d6aa9e9644fcc24482de85820bfe2ad341, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: c6e1c2339d66267100fd9c9851f6fb488e0e91054519c789bc30206a8bf0f175, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: ea6612af21a179810783dc4c8f39584a5f287099d46855b0b44b3d706ef349f7, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 010107031fdf6e54fcac08ac186ff4f0a9018d887bf5920f921f14e80bd82633, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: d3d910189e70e5e5edd9a3f76420da1e8a5578b966ceef1c3936fb6b8e456551, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 090b4fb6f42c01d28d3dd382ee3b06c9596078e719d863eb53fe16bc1a0ca910, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: fb965026ca12eae296ee0e5a21c0f7f2691e7f27a829c7d6b69b3d659257222a, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 602d69fbe97264184593d70751ac8421a674ccc32daa16d0a251266d600e41b6, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 710e9427be77576c22d1e45a519a3e25c804d22150ca272b321b7c6916bb08c7, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 901488c5b80e759ac79c75455b21b7680ecd546d5529c76483a65dd96a3dab82, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 2a096cf1746e2653f40f60879ceed7f994e888a06a029a143dc3b160040e756f, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 5f2869cd7e61c776c9b5ceb1ee3f92fd09cb1b636d76bf32e573d6a8a8faced0, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 25461ebb72e6daae6d0dc5815dcde9cd3fb2cf5150fe435f336775b00b45336d, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: a435d97ba2ea930c89581870ce3bdc86e96e6399ce7fc3ef7ed19bf88f3771a6, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 3a3c5b4774627ce2884a00d76ebda25faae4b9ac1e76da7ae81513a08531af21, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 71fb9aae0aa1fd2fd7b0a41ffa1c2235cd23ab372bb1d1d039e7ebf150aad656, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 178c1f72e05a48d00980e3f21a6a33eb66c2c5fd87d78b72a92d4423cf3b3e40, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 842cec572ddb0d7d642abdc3919a6b340a6787b4128d37184ad9d69095bdf875, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 9ed22ed7d6c73a08f9522684d4996821054dde714067e644bf8225fe9f2817ff, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 6674aadb0c124d37c4b10b3a8fb1fef68aa6e697c6c8b315f07244721921136f, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: cc82520bd9c7eeb5f06c9f7ebf1dd59b89bfd90d91080f88d49bd069250152e5, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: ae6113bfce471464f03e7ff173b9a9e13a8bb431439c41d82100097c5d61dd7d, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 5d098d85a099630e19fe3b715500589f0face5dafeffdf9cb3c23f5259600aa3, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: cea94c94a9f19381ae78825923b0c72cf9f16907bd0213bea6beca953a70b085, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 43b7ccbcb6eef8606b1b0aaf4c1c858948f75df49caa0589f74a107a9eaf1ea8, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: bb6b6f851602827f90538249a5eb7dffa6755060c85a51cd16b13500bbaf57d1, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 2c40553d646d1f2d657f5f982f49b9ca64dfd6e1b675965830c419412c6076c1, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 2cf1d19bf1e2c876de558ca796e1f48a65342d6927b11970b750152085d50ec7, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 9c74edf111b76d22e29f59efc87f353419396850fea9cecb6d1f3535d7370cea, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 537b9002304148b4aef6f995c3012d9ae159196ab216bfb6fa7ef40f5585cddb, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 7a5f7b3b6f5972d4a6c9a42a1e3f6cd73e2f8cb7cb57685a964ff9d6cd43a03d, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: aaa722230f998bcba4bfe53b5843b770a09c16203c4de187c1b810c8167b6471, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
question: 58223a1f18c3cda82967cc3ba7d24813209e41470aee583167ee938ae01d3d21, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 0e035026e0e096f6275c4e0699603f502f04d6c9904bc938b46df4dc6300116b, turn: 1, model: claude-3-5-sonnet-20240620, score: 0,
question: 1845745b3354782800cff1a055131c3e64c58719dced8d96d69083ab210e0391, turn: 1, model: claude-3-5-sonnet-20240620, score: 1,
100%|█████████████████████████████████████████| 50/50 [00:00<00:00, 9660.73it/s]
models: dict_keys(['claude-3-5-sonnet-20240620'])
Stats:
{
"bench_name": "live_bench/data_analysis/tablejoin",
"model_list": [
"claude-3-5-sonnet-20240620"
],
"total_num_questions": 50,
"total_num_matches": 50,
"output_path": "data/live_bench/data_analysis/tablejoin/model_judgment/ground_truth_judgment.jsonl"
}
0%| | 0/50 [00:00<?, ?it/s]question: 7c99777b2f4c5a9c88cc1f04d0345ac7b1e9dea2c7ac74b3fbf683e59bbf38f4, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: 7d3b232a7df622492efaa9230b09fe5a5e45c12d35ed346a99b6ec201497a1e3, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.67,
question: d89584191190995d5cb7307c938dbfb201e3af17ed7f666c2afae0fe2ad55985, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.29,
question: 1620e3381c6b9ba1ff0bcde15d816ec23ce445e1de6ed45de56ca41b0d1ae855, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: 01fc14e123214c67cbf235824d1ec952a825d5f78464ecc18fb9609c2781f50c, turn: 1, model: claude-3-5-sonnet-20240620, score: 1.0,
question: 490dfdc0383f199c870aa7710499c4081c35ff3545415dab3904f64e7526a809, turn: 1, model: claude-3-5-sonnet-20240620, score: 1.0,
question: 0764131eaf30bb8af36ad749f144da01c0113b1cee00092dde2919287df2ba78, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.57,
question: 55d610b0b74c049e9664df825f1bffcb7999fffc0576ff3317960a2124c3feaf, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.86,
question: 9d53b3ca366bedc7b149a5d41a4dc5c52cd76f1989a0cb6020d304fef6eb8d8d, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.5,
question: d4b2efd567053821eedf1ea3f759d4948f50264b94bd6ff37b18bc92e79d4fc1, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.24,
question: d60522bc74ae4e6d7ba1a5e0401e53e4a3d7a7182fed328e72825445ceafba9d, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.4,
question: e824359153d4fea96a9257ecceb44a3bb95dd0c84f95e2e3964ebdcdf8e8b32b, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: 519653e1054c2c48e303e4f8fb1fa2e5fe01d1fd1fb4d26fa45a33b5eb781a3c, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: a783dc9652728632d05f85ac5f944f71ffdfb2cc9dc6ea27e21ad80a96f44e48, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.2,
question: 4d351c29bdddf5c41d59cd7bd1b70bb4d2ae2a071ada382d7690066b1cd7764c, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.6,
question: 44953ce33916e7caae16bbce54fbd5a4e00d438924e5e53c0b5c5765ce5a583f, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.5,
question: a9622ef291b2ff5dac8ee5335d50d52a7bc8bd9fa001130fabaf3ae3d1505100, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.5,
question: 0bf086ff674cfda54c0293a3ae03a3720d2d1cb755748cc4800d43b375d20a3c, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: dd7ff515b9cd4c4a6e1d3fe3cb5e14c77123225c73193ce89c104b4f3f80cf22, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.67,
question: 52b2630e360ae523378662c58b554046d5086033761e830cee61d24e46850889, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.29,
question: a215b90180b104679133c979614fe0feeb770b6a3d1df4d41065e15be2ed7051, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.67,
question: d03bcee55bda5e582cc13547ab9bf898fbd1324fd5690481cc0d8a4ae9fd24f9, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.73,
question: b8a3e0f6c177bbef546e0dd490a0193b02124e193d5ffe093d86963449cba596, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: 2f1500d37ffd0e42cd2c89c04011cbbf5dd6b1f71f495156b016a967270cdded, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.77,
question: b2c9accaab7ee5cac67f482c19dcda8942fb409b25b604ef1136367f56d07fd0, turn: 1, model: claude-3-5-sonnet-20240620, score: 1.0,
question: 9318064da8b360eff10f17cdbde9ee624a2112203d8239516e536a0e5bec44e9, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.67,
question: 04ba0a2b8fe86cdd255723961356723f6de221cbe6bbc7af4b9ac93d45cd40ec, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.89,
question: 145cfcc10c148be13cc52c96a77611ff6fa5a2b2f756b7f8f9bc0220404a83d7, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: 1555bac3606cf98dc257767598c8a85738893f74b07a0a7f2d150751d0ab4939, turn: 1, model: claude-3-5-sonnet-20240620, score: 1.0,
question: fd0046f3c752ad7a6ce735aff42247b449563c3c664852793c698369c0046c93, turn: 1, model: claude-3-5-sonnet-20240620, score: 1.0,
question: 31b308131501939d06a5af26b6e26500ab71fc1585a16324abda514a2276ed14, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: 27da7f0ed5df368fa2d311fe3be17bbece8769109b41fc6e7768706d5d26f662, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.25,
question: eeec6c1afcb16c44895a770343d4c21c6eb88d2902ac8dc1568a6940d9502610, turn: 1, model: claude-3-5-sonnet-20240620, score: 1.0,
question: cb29bb1e6915d8366ff58783e47c9939d3d30712f2643cd23d6cbecc4210a2b2, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: 2e645a9a481f16ce14b5d069b62520852babd3b55383e00a75f675707088fddc, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.36,
question: 539fd06729e1f852302dd51aab15ffa115225362425ef04808cdef88d000d300, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.17,
question: a50e16a7dec04c766f864754305d6b28a99fe54602c7c913c525c067c405d279, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.8,
question: 75fca1a433c6e663241c1941e6034cd7625cd4b5981159c7f4ad74703df98b53, turn: 1, model: claude-3-5-sonnet-20240620, score: 1.0,
question: 140b7ab87b7be33e80fff3cfc052077d34cc51b5038c1c390cfb9780ad948c04, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.2,
question: 5063b77b06647a10818a76a2feda884741860ca4ef5816ae4580babafea11fb0, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.33,
question: ac146c48d703160bded02521568583372fc6b10bdbd98f36f57fcff7d0790d10, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.4,
question: 10047d040ef1e563f1db3278979d56d1182617b3484c63ed53a388a0d006a7e4, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.55,
question: a8995a220d4b23e751dded30067eb09897b7269b0ec3632762c9e97d41b80c95, turn: 1, model: claude-3-5-sonnet-20240620, score: 1.0,
question: 8b842182b7cbb2b961d8cdc64a1b4b28aff1f8ed4f4dd3fb58e3533baa754043, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: dc753a46614f7f4d1c839d06ec864324f8b6142e30bf804dae6aae8b6eb91941, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.14,
question: 4840c0c5075383274db75d8610087c3a725f4be885832e5fa97a46933e7485ae, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.0,
question: da9f424fc770103fa6b2639920d84fd8be3c448031ed96d13b975289356f4a67, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.14,
question: ae4654298c694908b994dd999e784904f1c22e2978e6e958d71cf0e5d5ab5975, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.8,
question: 587e13e04d18246f787cc8d41da67701eb1343795150a63b1996c5ec8270b20e, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.14,
question: bd4b2031ad50538f365ac3312534d813fb7326fd90cf5056ac80b31d189cbb15, turn: 1, model: claude-3-5-sonnet-20240620, score: 0.4,
100%|█████████████████████████████████████████| 50/50 [00:00<00:00, 4762.25it/s]
models: dict_keys(['claude-3-5-sonnet-20240620'])
Stats:
{
"bench_name": "live_bench/instruction_following/paraphrase",
"model_list": [
"claude-3-5-sonnet-20240620"
],
"total_num_questions": 50,
"total_num_matches": 50,
"output_path": "data/live_bench/instruction_following/paraphrase/model_judgment/ground_truth_judgment.jsonl"
}
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data] Unzipping tokenizers/punkt.zip.
/root/LiveBench/livebench_env/lib/python3.10/site-packages/nltk/app/__init__.py:29: UserWarning: nltk.app package not loaded (please install Tkinter library).
warnings.warn("nltk.app package not loaded (please install Tkinter library).")
Traceback (most recent call last):
File "/root/LiveBench/livebench/gen_ground_truth_judgment.py", line 327, in <module>
gen_judgments(
File "/root/LiveBench/livebench/gen_ground_truth_judgment.py", line 218, in gen_judgments
scores = instruction_following_process_results(questions, model_answers, task_name, model_id)
File "/root/LiveBench/livebench/process_results/instruction_following/utils.py", line 23, in instruction_following_process_results
results = evaluation_main.evaluator(questions, model_answers, result_log_dir, model_id)
File "/root/LiveBench/livebench/if_runner/instruction_following_eval/evaluation_main.py", line 271, in evaluator
outputs.append(func(inp, prompt_to_response))
File "/root/LiveBench/livebench/if_runner/instruction_following_eval/evaluation_main.py", line 119, in test_instruction_following_strict
if response.strip() and instruction.check_following(response):
File "/root/LiveBench/livebench/if_runner/instruction_following_eval/instructions.py", line 229, in check_following
num_sentences = instructions_util.count_sentences(value)
File "/root/LiveBench/livebench/if_runner/instruction_following_eval/instructions_util.py", line 140, in count_sentences
tokenizer = _get_sentence_tokenizer()
File "/root/LiveBench/livebench/if_runner/instruction_following_eval/instructions_util.py", line 135, in _get_sentence_tokenizer
return nltk.data.load("nltk:tokenizers/punkt/english.pickle")
File "/root/LiveBench/livebench_env/lib/python3.10/site-packages/nltk/data.py", line 763, in load
resource_val = restricted_pickle_load(opened_resource.read())
File "/root/LiveBench/livebench_env/lib/python3.10/site-packages/nltk/data.py", line 667, in restricted_pickle_load
return RestrictedUnpickler(BytesIO(string)).load()
File "/root/LiveBench/livebench_env/lib/python3.10/site-packages/nltk/app/wordnet_app.py", line 664, in find_class
raise pickle.UnpicklingError(f"global '{module}.{name}' is forbidden")
_pickle.UnpicklingError: global 'copy_reg._reconstructor' is forbidden
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment