Clojure Parallelism, beyond futures

intro

• Leon Barrett
  • Wrote claypoole

example

• Continuous rank probability score
• model output produces a distribution of samples
• want to compare observed to our model
• fn crps
  • We map this crps fn for each timestep
  • we get an average

How future works

• Future is the basic building block of parallelism
• @ (deref) will block until the future is done
• future -> future call (fn @body)
  • (.submit clojure.lang.Agent/soloExecutor ^Callable f)
• Threads
  • discussion of threads running on cpu
  • operating system will schedule
  • why do threads switch?
    • (they need to read disk, network)
  • concurrency vs parallelism
    • concurrency can happen without parallelism
    • parallelism is where more than one thing happens at once
  • threads have overhead
    • memory
    • take time to start
  • we use thread pools
• clojure uses a single threadpool
  • Agent thread pool
  • shared for all futures and agents
  • “unlimited threads”
    • (max int threads)
  • idle thread lifetime of 60s
  • program will not exit while threads remain
    • they aren’t .setDaemon threads
• limitations of future
  • you will have problems if
    • your takss are small compared to overhead
    • if you want to control the # of concurrent threads
    • you expect exceptions to work normally
      • They don’t
      • They get rethrown as j.u.c. Exceptions

How pmap works

• pmap lazily runs the next ncpus + 3 items in futures
• review of the code
  • n (+ 2 (get available processors))
  • rets (lazy (map #(future (f)))
  • fs
  • step (fn step []) makes a lazy seq
    • takes a sequence of futures
    • seq will force the first element of the futures
  • If we start with 8 tasks, we’ll force evaluation of the first 4
  • keep work in the pipeline with fs
• it’s lazy, it needs to be driven
• it uses futures to do the work
• generates threads as needed
  • beware simultaneous pmaps
  • it’s wacky when there’s chunking (lazy seqs are chunked)
• runs roughly ncpus + 3 tasks
  • a slow task will stall it
• Same limitations of pmap
  • if your tasks are small compared to overhead
  • exception porblem
  • you really want to saturate the CPU

General caveat

• if you don’t force a pmap, no work gets started
  • needs doall

core.async

• CSP channels and coroutines
• reads like one flow
• avoids callback hell
• uses cooperative multitrheading
  • your coroutines are lighterweight than threads
  • core async will switch coroutines only when it interacts with a channel
• backed by a fixed-size threadpool
• mostly for concurrency, not parallelism
  • you shouldn’t block many of it’s threads
  • easy to wait on other work
  • you might want to use it to interact with worker threads
• You can get parallelism using pipeline
  • runs a transducer between two channels with parallelism n
  • pipeline-async and pipeline-blocking
  • exceptions will kill your coroutine

claypoole

• goal was to get as much parallelism as possible
• use thread pools to control parallelism
• you need to manage threadpools
• default is eager, not lazy
• output is an “eagerly streaming sequence”
  • looks like a seq, blocks on incomplete tasks
• doesn’t stall on slow tasks
• (doall pmap sleep(1-10ms rand)
• built in map is 5.6 ms/task
• claypool is 5.6ms / task / thread
• built-in pmap averages 7.7ms / task / thread
  • because sometimes it waits on the slowest
• streaming seqs can be chained
• cp/future cp/pmap cp/pfor
• unordered functions available
• lazy version avails
• if you run a faster pmap with a slower pmap beneath it, you grow a buffer between them.
• exceptions get re-thrown correctly
• eliminates chunking
• can do priority threadpools

reducers

• uses java’s fork/join pool for flexibility
• reducers does give us a parallel reduce

aphyr/tesser

• Avoids fork/join pool for perf
• useful for cpu-bound operations
• distributable on hadoop

Boot can build it

Intro

• Alan Dipert

@alandipert

• Micha Niskin

@michaniskin

• Adzerk

Why a new build tool?

• There aren’t ways to make small modules to encapsulate complexity.
• Builds are processes, not specifications.
• Most tools oriented around configuration instead of programming
• We’re programmers, we need to program builds

One build tool

• Made of small independent parts that each do one thing well
• Small things are only useful if composition is left to the user

What is boot

• Uses maven under teh hood
• We use it to build clojure and clojurescript

A common build workflow

• Java example
• boot command line
• boot javac -h
  • some docs
• boot -V
  • more docs
• boot -s src/ show -f
  • basically tree showing pipeline
• boot -s show -f -- javac -- show -f
  • shows .class file that has been compiled
  • each step creates a new immutable fileset
    • think ring middleware

more cmd line examples

• boot -s src/ javac -- pom -p boot-demo -v 0.1 -- jar -m boot.Demo -- install
• boot -d boot-demo repl
  • boots into a repl of that .jar file
• everything you can do at the command line should be possible in the repl
• (set-env! :source-paths #{"src"})
• (boot (javac) (pom :project `boot-demo :version "0.1") (jar :main `boot.Demo) (install)
  • same as running from command line

unix analogy

• unix -> process program -> text
• boot -> process task -> fileset

anatomy of a task

• Task constructor, accumulated state, middleware, handler
• deftask
  • middleware pattern, after processing, passes to the next middleware for more processing.

deftask example

(deftask build []
  (comp
    (javac)
    (pom :project `boot-demo :version "0.1")
    (jar :main `boot.Demo)
    (install)))

(boot (build))

;; composing with watch
(boot (watch) (build))

you can put this in a file

#!/usr/bin/env boot

(defn -main [& argv]
  (boot (build) ...)

• boot will look for a build.boot, and generate a main mthd for you.
• You can define tasks in build.boot, and use from cmd line

Fileset

• a little anonymous git repo
• real files underneath but 100% managed
• basis for classpath
• immutable
  • tasks need to be able to modify anything
• query API
• add, remove
• commit: mutates underlying files

New tasks

• Converting Java into Fortran
• in the build.boot file

(deftask upcase
  "Convert file text contents to upper case"
  [x extension EXT str "The file extension"]
  (let [dir (temp-dir!)]
    (with-pre-wrap [fileset]
      (empty-dir! dir)
      (upcase-files dir (files-by fileset extension))
      (commit! (add-resouce fileset dir)))))

invocation:

`boot upcase -x .java`

pods

• How we avoid “dependency hell”
• Isolated Clojure runtimes
• Each can have different dependencies
• Easy to create, run code inside of
• Some things can’t be passed between pods

repl pod

(def env (assoc boot.pod/env :Dependencies `[[org.clojure/clojure "1.5.1"]]))
(def pod (boot.pod/make-pod env))
(boot.pod/with-eval-in pod (clojure-version)) ;; "1.5.1"
(clojure-version) ;; "1.6.0"

Well I Wouldn’t Want To Make a *Dys*functional Game

Intro

• Morgan Mullaney

An exploration of lisp in game dev

• Conrad Barski’s Land of Lisp
• Abuse 1996
  • (engine in C++), small home-grown lisp dialect for logic and UI
  • defchar (register a character in the game, and a set of callbacks)

More lisp games

• Crash bandicoot
• Jak and Daxter
• Naughty Dog Studios
  • bought by Sony

GOOL and GOAL

• Crash used (GOOL)
• Jak Game Oriented Assembly Lisp (GOAL)
• typed function arguments
• rlet lets you write directly in assembly
  • you can bind a symbol to a direct register

Vendetta Online

• Server-side NPC AI (DELIVERATOR) written in common lisp
• The “bugs” travel around the universe in swarms
• A big part of the game is hunting them down, killing them for access to asteroids
• You could connect a REPL to talk to the live AI system
  • devs would play game master, scripted events
• You could add new bot behavior (callback hell)
• This system became unmaintainable due to scope creep
• So they re-wrote it in erlang, because they had re-invented actors

ChromaShift 2012

• Built in ClojureScript
• Uses entity-compoonent-system architecture
• Went on to work on LightTable

2x0ng 2013

• david o’toole
• Built on xelf, an emacs-inspired 2d opengl game in common lisp

xelf

• Is it a lang? an ide? A library?
• Described as emacs for games
• The ability to have a REPL that runs inside your game
  • you can display your code alongside your game

how do you draw graphics?

• Really down to the metal: OpenGL
  • Penumbra blog post
• SDL abstraction (Simple DirectMedia Layer)
• But these are grossly procedural and rely on mutating state
• You can hang a pretty picture in front of it
• Hide those away from your game and don’t interact with them directly

what have we learned

• Lisps are easy to write. Make a language tailored to your game engine.
• Use an entity component system
• Embed a REPL for development

Simulant in Anger

Intro

• Ryan Neufeld

MI-X

• Payments system like Apple Pay for a big irish company or something.
• MI-X adds “Value Added Services”
  • Loyalty, coupon, etc.

MI-X

• Microservice arch
• “Landing on Mars”
  • No prior on the ground experience
  • Likening it to curiosity mission
• Launch for 100s of stores, 1000s of tellers
• No second chances
• What is success for MI-X
  • speed of delivery
  • 3.5 devs avg
• Consistent speed
• Scalability

How did NASA test curiosity

• Simulating reality
  • High Temp
  • Low Temp
• Wanted this for MI-X

How?

• Siege
• AB
• httperf?
• Integration tests (Selenium)

Simulation testing

• Rather than define what a test will do, we define what we could do the framework will take it from there.
• Explore correctness
• Features + Qualities = Well-functioning system
• We want a tool that will validate those qualities

Simulant

• Framework for simulation testing
• A test in four paths
  • model
  • test
  • sim
  • validate

Model

• Create a directed model graph of states in a random walk

Test

• Take a random walk and sample that into a linear walk (stream of actions)
  • We can re-sample this in the future to recreate a sequence of steps
  • We can generate n number of streams of actions

Sim

• Spawn any number of processes, as threads on one machine, or multiple machines
• Writes that to an action log in datomic

Validation

• Most validations can be run as queries in datomic

Simulant and MI-X

• Gather a baseline of performance
• Incrementally improve

How long does it take

• 0 -> simple [1 week]
• 1 month to get to a full multi-user flow
• “clojure: hard to learn but it pays you back”
  • simulant is the same

How does it pay us back

• Finding #0
  • Did this ever work?
• The system thrashed badly right away
• Auto scaling triggers via CPU
  • They were IO bound
• 5-10 users were making the system fall over
• create-payment endpoint was ~10s or never
  • 10 requests acq -> 10 requests tx -> 10 requests to auth

JWTs

• encrypted payloads as a token in JSON

A bunch of benchmark stuff

• Just some slides about decreasing errors, increasing mean response time

Slides

• slides

Clojure and ClojureScript update

Intro

• Alex Miller
• David Nolen

ClojureScript

• Recap of what has been released recently.

What’s new in cljs

• Not small anymore
• 20k sloc

Speed

• Because we were focused on just getting it working, it wasn’t fast.
• Was slow because invocations re-compiled 10k sloc
• Now
  • Analysis caching
  • Avoid analysis altogether
  • AOTed cljs.core (inc. analysis, source)
  • Will start shipping AOTd artifacts
    • tools.reader, data.json are available AOTd

cljs.jar

• just needs java 8
  • better for beginners
  • better for broader ecosystem
    • clearer separation between compiler fundamentals & downstream tooling
• When you submit a bug, can you repro with cljs.jar

:foreign-libs

• things like jquery
• free from externs hell
• allows the community to package useful libraries
• CLJSJS now has 30+ popular packages

cljs.test

• official, works with compiler settings
• 16k assertions when we test cljs
• most of the functionality from clojure.test
• works under :optimizations :none

static Vars

• Porting clojure.test highlighted the need for supporting a subset of the Var abstraction
• supported now (subset)
• can get some compiler information
• provide the usual metadata

macro usage

• new convenient behavior if namespace and macro namespace share name and the namespace requires it
• library users can just require your library

incremental compilation

• parent-ns -> child-ns didn’t work
• recompile dependents works now
• huge for testing
• works even for cold builds
• :recompile-dependents to disable
  • may be annoying for figwheel users

:modules

• Shipping large clojurescript apps?
  • single build, but large
• google solved this in 2011
• advanced builds can be code split
• split production builds into optimal pieces
• nothing to do with commonjs, es2015 amd or whatever
• only about code splitting
• google closure compiler once again delivers
• changed top-level, everything is static
  • even though we have a large standard library
• clojurescript emission is now optimized for it, comes for free

Conditional reading

• only took us 3.5y to get to this
• official support for conditional reading via .cljc extension
• supports 3 platforms
  • clojureclr
  • clojure
  • clojurescript
• needs feedback

repls

• we didn’t spend as much time on repls as we should have
• they were wrong, woefully wrong
• repls should work, the way you’ve come to expect
• new repls
  • jdk 8 nashorn repl
    • approaching perf of javascript
    • hopefully better in jdk0 9
  • node.js repl
    • fastest option and reflective of browser javascript perf
    • dnolen used it to port test.check
• repl overhaul
• all repls support stacktrace mapping via source maps
• generic support
  • figwheel works
  • ambly repl for iOS

repls now conform to clojure.main design

• perhaps not the best design, but nREPL tooling expects it

ambly repl

• The .js core in iOS is really good
• if you want to target iOS, android and the web
• mike fikes already shipped an app
• ambly/Clojure hosted under the om org
• don’t need xcode
• foundation for react native integration
  • uses bonjour and webdav

What’s next?

• API namespaces
  • we want tooling to be better and stable
  • we want people to stop calling into the impl namespace
• if you want to build a tool, build it on cljs.build.api & cljs.analyzer.api
  • relying on this will stabilize things for everyone
  • team will dog-food these namespaces

optional bootstrap

• now that we do conditional reading, there’s almost nothing left to do
• atom-shell (building portable desktop apps)

ES2015

• Final draft is out
• exploding popularity of React also making CommonJS modules common in the ecosystem
• clojurescript can’t consume this stuff

“system cljs”

• support the popular module formats
• google closure saves our butt again
• leverage js parsing/analysis tools
• externs inference and auto-generation

Clojure

• Alex Miller

clojure 1.7

• Some bugs and regressions reported, most are resolved
• should move to RC in a few weeks

transducers

• Composable algorithmic transformations
• Can take operations like map, reduce, filter and create a transducer out of that
• One less arity, omit the input source
• can use with core.async channels
• you can use comp to get l -> r semantics, like ->>
• new function called transduce, similar to reduce
  • first arg is a transducer
  • next arg is a reducing fn
  • then an initial value
  • then a coll
• example of into
• If you combine these things with collections that can reduce themselves, you can save a lot of object allocation
• eduction will re-compute the transformation each time you use that eduction
  • when you don’t want to hold on to the head and intermediate state in memory

Reader conditionasl

• Portable code across Clojure platforms
• .cljc extension
• reader conditionals

;; Choose an expression based on Platform
#?(:clj (java.util.Date)
   :cljs (js/Date.))

;; splicing version
[:a :b #?@(:clj  [:c :d]
           :cljs [:e :f])]

• You can fall through this to emit nothing (not nil)

Performance

• faster symbol and keyword construction
• reduce compile times (reduced class lookups)
• repeat, cycle, iterate, and range
  • faster sequence and reduce usage
• hash-maps and hash-sets now have direct iterators that don’t use seqs
• keys, vals faster iteration with reduced allocation over maps and sets
• iterator-seq is now chunked
• vec, set - significantly faster for most inputs
• partial - unrolls more args
• multimethod default value dispatch cached

Purely Random

• Creating random splittable RNGS

Use in test.check

• Sequence of numbers, and one from that sequence
• non deterministic impl in test.check

Testing random number generators

• dieharder
• Linear testing approaches
  • fibonnaci

Dieharder results

• j.u.random is linear and pretty weak
• results for different types using SHA1 technique

Performance

• SHA1 does 1k arithmetic ops, it can be slow
• Try a faster (noncrypto) psueudorandom function, test it’s quality

java.util.SplittableRandom

• Mutable, splittable random
• nextLong()

Algorithm

• Two “mixing” functions
• input size is 64bits long
• inputs in inner circle, outputs in outer circle
• seed for initial state
  • pass 24 to the constructor, the initial state is 24
• adds a special gamma number, gamma is odd
• pass in something from the mixing function, we get a new number
• gamma gets mutated inside

deftype IJUSR

(deftype IJUSR [^long gamma ^long state]
  IRandom
  (rand-long [_]
    (-> state (+ gamma) (mix-64)))
  (split [this]
    (let [state1 (+ gamma state)

Benchmarks (criterium)

• xoring 1,000,000 random numbers
• 150ms
• the sha1 impl is much slower
• diehard results are just as good

Wrapup of this section

• Linear RNGs can not be trivially splittabilized
• recent research provides promising options

Back to test-check

• test-check 0.8.0-ALPHA is out now, with this RNG
• Slowdown on it’s own test.check suite ~16.3% slower

But!

• Can parallelize test
• resuming shrinks
• parallelized shrinks
• custom shrinking
• generating lazy seq
• replay a specific test with a seed

Staying SAFE: A Foundation for Clojure Apps

• Ron Toland

Whatis Sonian

• Email archiving and search
• Clojure + Elasticsearch
• 2009

What is SAFE?

• Sonian Archive File Engine
• Application, Library, Framework -> Foundation

New Service, Old Problems

• How do I deploy it?
• How do I configure it?
• How do I debug it?
• How do I make it flexible?
• How do I run code on startup?
• How do I run periodic tasks?
• How do I give it commands?
• How do I distribute work?
• SAFE answers these

Config

• How do I configure it?
• Finds all the system configuration files on the classpath, merges them into a clojure map.
• a function config to pull values out of that map

Benefits

• Loaded in order
• can use to have defaults in one area
  • when running lein, the test directory will have a config
• different config for testing
• preserve sane defaults across services
• if SAFE’s config file is too large, you can break it up
• Open source carica

Modules

• They have to run on different systems, different scheduling.

Benefits

• config-controlled auto-require
• doinit and dofini
• ticks and tocks
• dostatus
• defadmins (spoilers!)

Open source version

• Carousel

Safectls

• command line to start a repl
• example command line invocations
• Some slides on writing these commands

Games and 3D Graphics in Arcadia

• Timothy Gardner, Ramsey Nasser
• Unity + Clojure

Unity

• Industry standard video game engine
• From HighSchool to AAA studios

What do I get?

• Graphics
• Physics
• Networking
• Entity Component System
• Authoring Tool
• Multiplatform

Clojure CLR

• MS$ runtime for C#
• David miller
• To make arcadia to work, we had to fork the compiler.

Dive through Arcadia

• Low level
• Not cute
• Tame Unity, get out of your way
  • Perlin noise always returns a const ?
• Libraries
• 1 year old

Examples

• Feminist side scroller
• Jongelars
  • Reads books from gutenberg, and puts them in books in 3d
• “An evening of modern dance”
  • If the physics freak out, you need to practice dancing more”
• Wizard hands

Code demo

• Functional game development process
• Entity compnent system in Unity
  • Every object in
• Arcadia can be a folder that you just add to unity
• Player as data

(ns demo.game
  (:use [acradia.core
        arcadia.linear])
  (:import [Vector2]))
  
(def ship-initial
  {:position Vector2/zero})

(def move [obj offset]
  (update obj :position v2+ offset))

(-> ship-initial
  (move (v2 3 1)))

• To get this to show up in unity, you need a component.

(defcomponent PlayerShip [state]
  (Start [this] (set! state ship-initial)))

The guts of Unity are scary

• Unity doesn’t actually use C#, it uses C++ and a Mono scripting engine

Transform component

• PlayerShip needs to talk to TransformComponent to move the ship

(def update-player [obj]
  (-> obj
      (move (Input/GetAxis "horizontal") ; -1 1
            (Input/GetAxis "vertical")))); ; -1 2

(defcomponent PlayerShip [state]
  (Start [this] (set! state ship-initial)
  (Update [this] (set! state (update-ship state))

• To reflect the change

(defcomponent PlayerShip [state]
  (Start [this] (set! state ship-initial)
  (Update [this] (set! state (update-ship state)
                 (set! (.. this transform position)
                       (v3 (.x (:position state)))
                           (.y (:position state))
                           0)))))

Upcoming arcadia features

• Procedurally generated 3d blocks
• Paying for object instantiation in Unity is expensive
• For the next release
  • Stability
  • Web export
  • Docs
• hydrate
• To turn state into clojure data functions
• Brandon bloom’s dispatch-map library?

Everything will Flow

• Zach Tellman

Talking about Queues

• Producer -> Consumer
• FIFO
• Non-obvious part is that enqueueing things is a side-effect

Core async

• 3 queues are under the hood
• producer -> puts -> buffer -> takes -> consumer
• Consumer is provided a callback

More queues

• Has to be run on a threadpool
• java.util.concurrent.Executor
• BlockingQueue
• RejectedExecution When the queue is full

Conditioned queues

• not empty
• not full
• producer -> notFull -> buffer -> notEmpty -> consumer

java.util.concurrent.locks.Conditino

• software threads
• hardware threads
• probably like half dozen queues

Queues separate the “What” from the “when”

• a great property for systems

Queueing theory

• There is a lot of math
• You can ignore most of it
• Presenting here the things that intersect between systems
• Book Mor Harchol Balter
  • Performance modeling and design of computer system
  • Queueing theory in action

Closed systems

• Key distinction between closed and open systems
• A REPL is a closed system
• A single person browsing a website is a closed system

Open systems

• Where there is no coordination
• Almost guaranteed that a request will come in before we’re ready

A Simulation of an open system

• Producers are exponentially distributed
  • l = 0.5, l = 1, l = 1.5
• Consumers are Pareto Distributed
  • a = infinite
  • a = 3
  • a = 2
  • a = 1
• The 80/20 rule
  • How often unusually complex tasks arrive

Simulation part 2

• In the simulation, we’ll use a spectrogram
• Vertical axis is latency (log10)
• Horizontal axis thin tail -> fat tail
  • most tasks are easy, some tasks are complex in the fat tail
• Heat map showing where most of our tasks are in red
• A system in crisis, when our queues are out of control
• The time being spent on the queue dominates your time
  • Everything averages out, consistent but slow
• We know when a system is out of resources

Adding more consumers

• Showing a graph where 1, 2, 4 consumers keep up
• Next slide showing same failures for 1 4 and 16 producers and consumers
• What they look like as they lead up to failure, they fall over at the same place
• We can have a system that responds in sub-ms time, then falls over

What have we learned?

• Your system is probably open
• It’s the long tail that makes queues fill up
• more consumers make us robust to variance
• once the system falls, it falls hard
• unbounded queues are fundamentally broken
  • puts the correctness of your system in someone else’s hands

How do you salve a problem like too much data?

• 3 strategies
  • drop the data
  • reject the data
  • pause the data

Drop

• only valid if newer data obsoletes older data of if we just don’t care
  • telemetry
• or if we’re just doing best effort
• We do this too readily

Reject

• often the only choice for an application
• when we have too many requests, we often time have no better method
• Need to understand what we’re going to do in this situation

Pause

• AKA Backpressure
• Often the only choice for a closed system, library or sub-system

Backpressure swims upstream

• producer -> [puts] -> [buffer] -> [takes] -> consumer ^^ backpressure here

Queues, what are they good for

• separating what from when

Default channel config in core.async

• No buffer by default
• Closed system

Why would we want buffers?

• Backpressure ain’t free
• restarting threads
• restarting tcp traffic
• reading from disk
• reading from databases
• buffers allow us to stabilize throughput at the expense of latency

Always plan for too much data

• What’s our response?
• Most neutral response is backpressure
• Flow up on a chain to a periphery
• that logic should be centralized
• most queues shouldn’t have buffers
  • We should avoid chaining buffers

What we know vs what we’ve done

• A lot of the attempts to use core.async, which has backpressure
• EE spec for websockets doesn’t have backpressure
• library authors haven’t really kept this
• we want to make sure that we don’t have to rebuild

Concrete example

• MAW
• Arbitrary volumes of data
• persist it somewhere for batch processing
• Did this on AWS
  • ELB -> machines -> s3

Key properties

• Horizontal scale
• Low maint
• not real time
• loss of data ~ loss of utility

Previously

• 600 tweets/sec -> http client -> hadoop s3 client -> s3
• Adapted this for MAW
  • 20,000 -> aleph -> hadoop s3 client -> s3

s3 stopped accepting writes

• Hadoop s3 client held all this in memory, ran out and fell over
  • cascading failure
• Wasn’t a failure mode I was thinking about

Ideally

• -> Aleph -> [] -> [?] -> s3
• I just wanted the data to be somewhere other than memory
• Could use local disk

Two libraries

• Factual/durable-queue
• Factual/s3-journal
  • built on top of durable queue

And so

• Aleph -> s3-journal -> s3 [ durable queue]
• The problem with the hadoop s3 library was that the queue was invis

Metrics within durable queue

Netty’s threading model

• We have to get back to each of those requests
• Every time a connection is opened, it’s assigned to a thread
• Every time a connection blocks, it blocks that thread
• Netty adds a BlockingQueue
  • Threadpools in here are a little invis

Metrics?

• ztellman has a library to inspect threadpools
• dirigiste
• task-rejection-rate
• task-completion-rate
• task-arrival-rate
• task-latency
• queue=latency
• queue-length
• num-workers

Sampled stats, not a perfect representation.

What we’ve learned

• What does completion mean?
• Be picky about your tols
• Prefer metrics to raw performance
• You’re never measuring all of your system

An abstract example

• A chat server

Manifold

(require
  '[manifold.deferred :as d]
  '[manifold.stream :as s]
  '[manifold.bus :as b])

(def chat-handler [conn]
  (d/let-flow [room (read-room conn)]
    (s/consume

• If anyone is slow on the connection, it blocks everyone
• want to make sure that one bad actor doesn’t slow everything

Publishing

• Maybe with message sized buffer?

(s/connect
  (p/subscribe bus room)
  (s/buffer msg-len 1e3 conn))

Source -> Sink

• with timeout
• If I have been trying for 10s, you’re done, you can reconnect

What if someone is publishing too fast?

• Being a jerk and being naieve, you can’t distinguish
• It’s your job to make sure that you don’t get knocked over
  • That the many don’t suffer due to the actions of the few

Throttling

• 1msg/sec
• Failure is on the publisher
• manifold example to throttle with credit

Understanding dynamic topologies

• We have an untold number of queues in the chat example
• If you have core.async shoving messages everywhere
• In manifold, you can look at the topology, look at each of the streams
• Unclear how to visualize these
  • Open topic
• In the clojure ecosystem, we need to talk about this

We talked about queues

• unbounded queues aren’t
• the application should plan for too much data
• elsewhere, defer to the application
• demand metrics from everything