- Proposal: SE-NNNN
- Authors: Chris Lattner, Dan Zheng
- Review Manager: TBD
- Status: Awaiting implementation
This proposal is a follow-on to SE-0195 - Introduce User-defined "Dynamic Member
Lookup" Types
which shipped in Swift 4.2. It introduces a new @dynamicCallable attribute, which marks
a type as being "callable" with normal syntax. It is simple syntactic sugar
which allows the user to write:
a = someValue(keyword1: 42, "foo", keyword2: 19)and have it be interpreted by the compiler as:
a = someValue.dynamicallyCall(withKeywordArguments: [
"keyword1": 42, "": "foo", "keyword2": 19
])Many other languages have analogous features (e.g. Python "callables", C++ operator(), and
functors in many other languages), but the
primary motivation of this proposal is to allow elegant and natural interoperation with
dynamic languages in Swift.
Swift-evolution threads:
- Pitch: Introduce user-defined dynamically "callable" types.
- Pitch #2: Introduce user-defined dynamically “callable” types.
- Current pitch thread: Pitch #3: Introduce user-defined dynamically “callable” types
Swift is exceptional at interworking with existing C and Objective-C APIs and we would like to extend this interoperability to dynamic languages like Python, JavaScript, Perl, and Ruby. We explored this overall goal in a long design process wherein the Swift evolution community evaluated multiple different implementation approaches. The conclusion was that the best approach was to put most of the complexity into dynamic language specific bindings written as pure-Swift libraries, but add small hooks in Swift to allow these bindings to provide a natural experience to their clients. SE-0195 was the first step in this process, which introduced a binding to naturally express member lookup rules in dynamic languages.
What does interoperability with Python mean? Let's explain this by looking at an example. Here is some simple Python code:
class Dog:
def __init__(self, name):
self.name = name
self.tricks = [] # creates a new empty list for each dog
def add_trick(self, trick):
self.tricks.append(trick)With the SE-0195 features introduced in Swift 4.2, it is possible to implement a Python
interoperability layer
written in Swift. It interoperates with the Python runtime, and project all Python values into a
single PythonObject type. It allows us to call into the Dog class like this:
// import DogModule.Dog as Dog
let Dog = Python.import.call(with: "DogModule.Dog")
// dog = Dog("Brianna")
let dog = Dog.call(with: "Brianna")
// dog.add_trick("Roll over")
dog.add_trick.call(with: "Roll over")
// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog.call(with: "Kaylee").add_trick.call(with: "snore")This also works with arbitrary other APIs as well. Here is an example
working with the Python pickle API and the builtin Python function open. Note that we
choose to put builtin Python functions like import and open into a Python namespace
to avoid polluting the global namespace, but other designs are possible:
// import pickle
let pickle = Python.import.call(with: "pickle")
// file = open(filename)
let file = Python.open.call(with: filename)
// blob = file.read()
let blob = file.read.call()
// result = pickle.loads(blob)
let result = pickle.loads.call(with: blob)This capability works well, but the syntactic burden of having to use
foo.call(with: bar, baz) instead of foo(bar, baz) is significant.
Beyond the syntactic weight, it directly harms code clarity by making code hard to read
and understand, cutting against a core value of Swift.
The @dynamicCallable attribute in proposal directly solves this problem.
These examples become their natural and clear form, and effectively match the
original Python code in expressiveness:
// import DogModule.Dog as Dog
let Dog = Python.import(“DogModule.Dog")
// dog = Dog("Brianna")
let dog = Dog("Brianna")
// dog.add_trick("Roll over")
dog.add_trick("Roll over")
// dog2 = Dog("Kaylee").add_trick("snore")
let dog2 = Dog("Kaylee").add_trick("snore")Python builtins:
// import pickle
let pickle = Python.import("pickle")
// file = open(filename)
let file = Python.open(filename)
// blob = file.read()
let blob = file.read()
// result = pickle.loads(blob)
let result = pickle.loads(blob)This is a proposal is purely syntactic sugar - it introduces no new semantic model to Swift at all. We believe that interoperability with scripting languages is an important and rising need in the Swift community, particularly as Swift makes inroads into the server development and machine learning communities. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well (e.g. for implementing dynamic proxy objects).
We propose introducing a new @dynamicCallable attribute to the Swift language which may be
applied to structs, classes, enums, and protocols. This follows the precedent of
SE-0195.
Before this proposal, values of these types are not valid in a function call
position: the only callable values that Swift has are those with a function type (functions, methods,
closures, etc) and metatypes (which are initializer expressions like String(42)). Thus,
it is always an error to "call" an instance of a nominal type (like a struct, for instance).
With this proposal, types that adopt the new attribute on their primary type declaration become "callable" and are required to implement one or more methods to implement the call behavior.
To support these cases, a type with this attribute is required to implement at least one of the
following two methods. In the examples below, T* are arbitrary types, S*
must conform to ExpressibleByStringLiteral (e.g. String and StaticString).
func dynamicallyCall(withArguments: [T1]) -> T2
func dynamicallyCall(withKeywordArguments: [S : T3]) -> T4We write Arguments as an array type and KeywordArguments as an dictionary type, but these
are allowed to be any type that conforms to the ExpressibleByArrayLiteral and
ExpressibleByDictionaryLiteral protocols, respectively. The later is inclusive of
DictionaryLiteral which
can represent multiple instances of a 'key' in the collection. This is important to support
duplicated and positional arguments (because positional arguments have the empty string ""
as their key).
If a type implements the withKeywordArguments: method, it may be dynamically called with both
positional and keyword arguments (positional arguments have the empty string "" as their key).
If a type only implements the withArguments: method but is called with keyword arguments,
a compile time error is emitted.
Because this is a syntactic sugar proposal, additional behavior of the implementation
methods is directly expressed: for example, if these types are defined to be throws or
@discardableResult then the corresponding sugared call is as well.
Since there are two @dynamicCallable methods, there may be multiple ways to handle some
dynamic calls. What happens if a type specifies both the withKeywordArguments: and
withArguments: methods?
We propose that the type checker resolve this ambiguity towards the tightest match based
on syntactic form of the expression. If a type implements both the withKeywordArguments: and
withArguments: methods, the compiler will use the withArguments: method for call sites that
have no keyword arguments and the withKeywordArguments: method for call sites that have at least one
keyword argument.
This ambiguity resolution rule works out very naturally given the behavior of the Swift type checker, because it only resolves call expressions when the type of the base expression is known. At that point, it knows the capabilities of the type (whether the base is a function type, metatype, or a type where one of these two methods exist) and it knows the syntactic form of the call.
This proposal does not require massive or invasive changes to the constraint solver. Please look at the implementation for more details.
Here, we sketch some example bindings to show how this could be used in practice. Note
that there are lots of design decisions that are orthogonal to
this proposal (e.g. how to handle exceptions) that we aren't going into here. This is just to
show how this feature provides an underlying facility that authors of language bindings can
use to achieve their desired result. These examples also show
@dynamicMemberLookup to illustrate how they work together, but elides all other the
implementation details.
JavaScript supports callable objects but does not have keyword arguments.
Here is a sample JavaScript binding:
@dynamicCallable @dynamicMemberLookup
struct JSValue {
// JavaScript doesn't have keyword arguments.
@discardableResult
func dynamicallyCall(withArguments: [JSValue]) -> JSValue { ... }
// This is a `@dynamicMemberLookup` requirement.
subscript(dynamicMember member: JSValue) -> JSValue {...}
// ... other stuff too of course ...
}On the other hand, a common JavaScript pattern is to take a dictionary of values as a stand-in
for argument labels (called like example({first: 1, second: 2, third: 3}) in
JavaScript). A JavaScript bridge in Swift could choose to implement keyword argument support
to allow this to be called as example(first: 1, second: 2, third: 3) from Swift
code (kudos to Ben Rimmington for this observation).
Python does support keyword arguments. While a Python binding could implement just the
withKeywordArguments: method, it may be better to implement both the non-keyword and keyword forms
to make the non-keyword case slightly more efficient (avoid allocating temporary storage) and to make
direct calls with positional arguments nicer (x.dynamicallyCall(withArguments: 1, 2)
instead of x.dynamicallyCall(withKeywordArguments: ["": 1, "": 2])).
Here is a sample Python binding:
@dynamicCallable @dynamicMemberLookup
struct PythonObject {
// Python supports arbitrary mixes of keyword arguments and non-keyword
// arguments.
@discardableResult
func dynamicallyCall(
withKeywordArguments: DictionaryLiteral<String, PythonObject>
) -> PythonObject { ... }
// An implementation of a Python binding could choose to implement this
// method as well, avoiding allocation of a temporary array.
@discardableResult
func dynamicallyCall(withArguments: [PythonObject]) -> PythonObject { ... }
// This is a `@dynamicMemberLookup` requirement.
subscript(dynamicMember member: String) -> PythonObject {...}
// ... other stuff too of course ...
}Following the precedent of SE-0195, this attribute must be placed on the primary definition of a type, not on an extension.
This proposal does not introduce the ability to provide dynamically callable
static/class members. We don't believe this is important given the goal of
supporting dynamic languages like Python, but it could be explored if a use case is discovered
in the future. Such future work should keep in mind that call syntax on metatypes is already
meaningful, and that ambiguity would have to be resolved somehow (e.g. through the most specific rule).
This proposal supports direct calls of values and methods, but subsets out support for currying methods in Smalltalk family languages. This is just an implementation limitation given the current state of currying in the Swift compiler. Support can be added in the future if there is a specific need.
This is a strictly additive proposal with no source breaking changes.
This is a strictly additive proposal with no ABI breaking changes.
This has no impact on API resilience which is not already captured by other language features.
In addition to supporting languages like Python and JavaScript, we would also like to grow to support Smalltalk derived languages like Ruby and Squeak. These languages resolve method calls using both the base name as well as the keyword arguments at the same time. For example, consider this Ruby code:
time = Time.zone.parse(user_time)The Time.zone reference is a member lookup, but zone.parse(user_time) is a method
call, and needs to be handled differently than a lookup of zone.parse followed by a direct
function call.
This can be handled by adding a new @dynamicMemberCallable attribute, which acts similarly to
@dynamicCallable but enables dynamic member calls (instead of dynamic calls of self).
@dynamicMemberCallable would have the following requirements:
func dynamicallyCallMethod(named: S1, withArguments: [T5]) -> T6
func dynamicallyCallMethod(named: S2, withKeywordArguments: [S3 : T7]) -> T8Here is a sample Ruby binding:
@dynamicMemberCallable @dynamicMemberLookup
struct RubyObject {
@discardableResult
func dynamicallyCallMethod(
named: String, withKeywordArguments: DictionaryLiteral<String, RubyObject>
) -> RubyObject { ... }
// This is a `@dynamicMemberLookup` requirement.
subscript(dynamicMember member: String) -> RubyObject {...}
// ... other stuff too of course ...
}This proposal is mainly directed at dynamic language interoperability. In this
use-case, it makes sense to take a variable sized list of arguments where each argument has
the same type. However, there are other use cases where it could make sense to support
static argument lists, akin to operator() in C++. For example, consider something like this:
struct BinaryFunction<T1, T2, U> {
func call(_ argument1: T1, _ argument1: T2) -> U { ... }
}It is not unreasonable to look ahead to a day where sugaring such things is supported, particularly when and if Swift gets variadic generics. This could allow typesafe n-ary smart function pointer types.
We feel that the approach outlined in this proposal supports this direction. When and if a
motivating use case for the above feature comes up, we can simply add a new form to
represent it, and enhance the type checker to prefer that according to the "most specific
match" rule. If this is a likely direction, then it might be better to name the attribute
@callable instead of @dynamicCallable in anticipation of that future growth.
Many alternatives were considered and discussed. Most of them are captured in the Alternatives Considered section of SE-0195.
Here are a few other points that were raised in the discussion:
-
It was suggested that we use subscripts to represent the call implementations instead of a function call, aligning with
@dynamicMemberLookup. We think that functions are a better fit here: the reason@dynamicMemberLookupuses subscripts is to allow the members to be l-values, but call results are not l-values. -
It was requested that we design and implement the 'static callable' version of this proposal in conjunction with the dynamic version proposed here. In the author's opinion, it is important to consider static callable support as a likely future direction to make sure that the two features sit well next to each other and have a consistent design (something we believe this proposal has done) but it doesn't make sense to join the two proposals. So far, there have been no strong motivating use case presented for the static callable version, and Swift lacks certain generics features (e.g. variadics) that would be necessary to make static callables general. We feel that static callable should stand alone on its own merits.