tmbb
Zee3 - Bindings to the Z3 theorem prover
I have written some bindings to the Z3 theorem prover. The Z3 executable is used as part of a port (i.e. I am not binding to the executable using the C API), which allows for interactive proofs using the Zee3.push(solver_pid) and Zee3.pop(solver_pid) functions. The bindings are heavily inspired by the Python bindings (link here), but with some Elixir twists: the main interaction happens inside a macro which allows us to pretend we are not interacting with a Port, while keeping everything outside of the macro very explicit. Technical note: the Z3 executable is downloaded during compilation of the package, but so far I have only tested it on Linux (Ubuntu 22.4). Downloading for other platforms may fail. You can always download the package for your system and set it as the executable using the config :zee3, z3_executable: "z3"
Hex package here and github repo here.
For those who don’t know what Z3 is, it is simply a magical (I don’t use the term lightly) program which allows you to define constants and constraints satisfied by those constants and will tell you whether the constraints are satisfiable or not. By encoding a property of the system (a program, a scheduling program, etc.) as a satisfiability problem, Z3 will tell you whether it’s true or not, and which assignment of constants will make it true. It does so by extending SAT solving with more complex theories pertaining to more complex objects such as Integers, Real numbers, Floating Point numbers (which respect the rules of floating point numbers instead of actual real numbers), Bit Vectors and Regular expressions.
It has some common “sub-engines” as a special case, such as a Simplex solver (which does the same as my Dantzig package as a special case) and a Datalog engine. I can seamlesly combine, for example, the Simplex solver with some more general constraints (which will obviously destroy the fast runtime of the simplex solver).
You can use this to find an input of any function f(x1, x2, ...) = y given y and the implementation for f. Note that Z3 may fail to find whether a formula is satisfiable (in this case, it returns unknown) or may take time which is exponential as a function of the length of the size of the problem.
The package still needs proper documentation, but I don’t antecipate anything to change in the public-facing API. Contributions are very welcome.
Examples
The best way to get a feeling for how it works, is to look at some of the tests.
Example 1
The comments explain the details of the code, but the basic idea is the following:
- We start the solver as a stateful inteval process with which we will communicate through message passing
- We create a
Zee3.programwhich allows us to add the declarations and the constraints, while seamlessly mutating the internal state of theZee3solver to add the appropriate variables - We
check_and_get_model!()to check for satisfiability and get the model (i.e. a set of variable assignments that satisfies the constraints) - We return the last expression in the program, which happens to be the value for
check_and_get_model!(), but it doesn’t need to be, like the next example will show
test "simple program" do
{:ok, pid} = Zee3.start_solver()
zero = 0
{:sat, model} =
Zee3.program pid do
# Declare integer constants
# Note: there is nothing special with the `Sort.int()` function call,
# the `Sort` module is just the `Zee3.StdLib.Sort` module, which the
# `Zee3.program` macro aliases inside the body so we can refer to it
# without needing to alias it ourselves.
a = declare_const("a", Sort.int())
b = declare_const("b", Sort.int())
# Declare a function that takes two integers and returns an integer
f = declare_fun("f", [Sort.int(), Sort.int()], Sort.int())
# Assert a constraint (note that Zee3 recognizes literal integers correctly)
assert f.(a, b) == -5
# Assert a new constraint, using a variable defined outside of the program
assert f.(a, b) + f.(b, a) == zero
# Note: the `assert/2` above is a Z3 assertion, not an ExUnit assertion.
# Check for satisfiability and get the model if satisfiable
check_sat_and_get_model!()
end
# "normal" assert, taken from ExUnit and unrelated to Z3
assert model["f"].(model["b"], model["a"]) == 5
# Note that the `model["f"]` is an actual anonymous function
# which can be called from Elixir like any other function
# (of course this function only takes "intersting" values
# in the values we assert in the program)
end
Example 2
What if we need to return two values from the program? That is absolutely not a problem, because internally the program retains the semantics of variable assignments so that we can assiign our results to variables and return them ast the end.
test "forgets constraints and variables whe popped", %{solver: solver} do
# We want to check for the satisfiability of two different
# constraint systems using the `push()` and `pop()` functions
# to add and remove new constraints.
#
# Because we have two different satisfiability results,
# we need to return the result of *two different*
# `check_sat_and_get_model!()`.
#
# This is absolutely not a problem, because the `Zee3.program`
# respects Elixir's semantics around variable assignment and
# we can simply assign the results to variables, or save them
# in a map, or whatevet, and simply return the results at the end.
{result_1, result_2} =
Zee3.program solver do
a = declare_const("a", Sort.int())
assert a > 0
# Create a new context here
push()
temp_b = declare_const("temp_b", Sort.int())
assert temp_b == a
assert temp_b < 0
# This variable is not made available outside the program,
# we will need to return it as part of the last expression.
result_1 = check_sat_and_get_model!()
# Pop the context that defined the `temp_b` variable
pop()
# As above, the variable is not made available outside
# the program
result_2 = check_sat_and_get_model!()
# Return the two results
{result_1, result_2}
end
# Refers to the new context
assert :unsat == result_1
# Refers to the original context
assert {:sat, model} = result_2
assert model["a"] > 0
refute Map.has_key?(model, "temp_b")
end
Example 3
Define variables dynamically using “normal” elixir constructs:
test "for loops work inside the program", %{solver: solver} do
{:sat, model} =
Zee3.program solver do
# Dynacmially create a number of variables and store them
# in a list. In this case, we create 10 variables, with
# names of the form x_i (for i in 1..10).
xs =
for i <- 1..10 do
# Note that there is nothing special about the first
# argument of `declare_const/2`, which can be anything
# that returns a string.
_x_i = declare_const("x_#{i}", Sort.int())
end
# Assert that they sum to 10.
# Where does the `sum/1` function come from?
# It's just a function that the `Zee3.StdLib` module defines
# and imports inside the program. You can easily define your
# own functions and use them inside the program, as long
# as the functions return the right format, as documented
# elsewhere
assert sum(xs) == 10
# Assert pairwise comparisons between all the variables.
# Note: there is actually a built in for this, but we really
# wanted to show that we can use for loops and normal Elixir
# functions without any issues
for {x_i, x_i_plus_1} <- Enum.zip(Enum.drop(xs, 1), xs) do
assert x_i == x_i_plus_1
end
check_sat_and_get_model!()
end
# Asert that all variables exist and are set to the only
# value that satisfies the given constraints
assert model["x_1"] == 1
assert model["x_2"] == 1
assert model["x_3"] == 1
assert model["x_4"] == 1
assert model["x_5"] == 1
assert model["x_6"] == 1
assert model["x_7"] == 1
assert model["x_8"] == 1
assert model["x_9"] == 1
assert model["x_10"] == 1
end
First Post!
mudasobwa
That sounds intriguing; I use z3 in Cure ( Type System — Dependently-Typed BEAM Language ) and I might maybe benefit from using this library instead of shelling out.
Most Liked
Asd
Thats a security and maintenance nightmare of every person responsible for dependency management.
First of all, I think that any library must never
- download executable binaries
- access internet during compilation
- download not hash-verified binaries
And the way this library does it is just very wrong.
First of all, it uses httpc without any authority verification on the host. So just changing the /etc/hosts file to something like github.com my.malicious.binary.website.com would still work if the website would just return 200 with a self-signed github cert over https. If attacker has can poison the DNS for the build machine, they can run an arbitrary binary in the production.
Second, it downloads the binary for the machine it is compiled on. Again, doesn’t work when you compile on x86-64 and run on x86 for some reason.
Third, no hash verification. If z3 repository gets hacked and some other binary is returned, you may never know, because you don’t compare the hash of the file you got to the hash of the file you expected.
Fourth, compilation will fail when internet or a github will be unavailable. Imagine that your build pipeline is failing 1% of the time, because that’s how often the github fails.
The solution is to remove the vulnerable code and everything regarding this download-during-compilation approach and leave the binary download instructions in the README.
You don’t need to implement the buggy and vulnerable package manager, there are lots of tested, efficient and secure ones in the wild. Like Nix, for example.
tmbb
I didn’t know anyone here was using Z3 for non-trivial tasks, it’s great to see that other people see the use of Z3 (and see the use of avoiding Python when possible).
If you are using Z3 as an interactive theorem prover (i.e. using (push) and (pop)) I think you can benefit. If you run a new instance of Z3 everytime you want to check something, I’m not sure what the real benefit is. I am currently adding an escape hatch to add literal SMT-LIB2 text to the running Z3 process (you can already do that using Port.command, but it’s not part of the public API). Feel free to contribute if you find this useful!
tmbb
Some updates on the implementation: I have added some major functionality which will probabbly make this package very useful for other applications.
Datalog implementation
I have added bindings to the Z3 datalog engine (called muZ), which despite not being very performant, it is much more performant than I can implement natively in Elixir, especially when one needs the extensions to standard datalog that Z3 provides.
require Zee3
test "simple program with string entities", %{solver: pid} do
{:ok, {:sat, solutions}} =
Zee3.program pid do
# Make sure the relations between entities declare entities
# with the right sort from the standard library.
edge = declare_rel("edge", [Sort.entity_id(), Sort.entity_id()])
path = declare_rel("path", [Sort.entity_id(), Sort.entity_id()])
# Declare variables, which, again, will use the correct sort.
a = declare_var("a", Sort.entity_id())
b = declare_var("b", Sort.entity_id())
c = declare_var("c", Sort.entity_id())
# Transitive closure definition
rule(path.(a, b) <- edge.(a, b))
rule(path.(a, c) <- path.(a, b) and path.(b, c))
rule(edge.(entity_id("alice"), entity_id("bob")))
rule(edge.(entity_id("alice"), entity_id("charlie")))
rule(edge.(entity_id("bob"), entity_id("dan")))
query("path")
end
# Z3 may return the tuples in any order so we sort them
assert Enum.sort(solutions) == [
{"alice", "bob"},
{"alice", "charlie"},
{"alice", "dan"},
{"bob", "dan"}
]
end
Finite sets implemented on top of existing theories
Small finite sets are quite useful in general, and it is useful to see how one can implement the basic axioms and operators of finite set theory in a relatively performant way in Z3. A lot of the operations are quadratic, so this is actually not super efficient, but I’m not sure I can do much better.
An example from the tests:
require Zee3
alias Zee3.StdLib.FiniteSet
test "declare a set of bit vectors", %{solver: solver} do
{:sat, model} =
Zee3.program solver do
s = FiniteSet.declare("s", Sort.bit_vec(8), max_size: 10)
assert FiniteSet.contains(s, "a")
assert FiniteSet.contains(s, "b")
assert FiniteSet.contains(s, "c")
assert FiniteSet.contains(s, "d")
assert s.cardinality == 4
check_sat_and_get_model!()
end
{:ok, s} = FiniteSet.extract_from_model(model, "s")
assert MapSet.equal?(s, MapSet.new(["a", "b", "c", "d"]))
end
Next steps
Another very useful feature of Z3 is a performant implementation of the Simplex algorithm for linear programming (optimization of a linear function under linear inequality constraints) and a not so performanc implementation of general non-linear optimizations. The current system can already handle this if one sets the right options (e.g. rounding the advanced real values, which are often presented as rationals and as roots of polynomials, which are pretty much impossible to calculate in Elixir), but I might want to add some wrappers around it.
Note that if anyone needs “normal” linear programming I already have a much better and much more performant solution in the form of the Dantzig package.
Last Post!
tmbb
Clearly we have very different definitions of what convenience is.
This is maybe because I am not a professional software developer and only write code to solve my own (and my business’s) problems (although I do work with some professinal devs for the more “professional” stuff), but the workflows you describe seem positively ghastly to me… As a possible consumer of this library, the idea that I would have to download any kind of software outside of what Mix already does for me, make sure it’s added to the path, etc. is a very, very big inconvenience. I envision this library as being useful for someone who wants to do some logic programming from Elixir, not necessarily someone who needs a super safe setup for production deployment (and you can get that safety once I implement the flags to avoid downloading the binary). The user I have in mind doesn’t need to know anyting about containerization.
To me, convenience is not having to leave the comfortable world of a mix.exs file and have everything work automatically and in which stuctured data is passed in a sane way between functions or sent as a message through processes without (god forbid!) ever having to use something like a unix shell where bash scripts receive and output somewhat free form plain text that is sent to some other bash scripts or weird unix utility that people insist on using as a library when the API is actually much closer to that of a command line application. To you, using the OS outside the BEAM is convenience… To me, using the OS outside the BEAM and integrate it with the BEAM is the definition of inconvenience…
I do acknowledge your arguments, and I will work to make the applications usable without downloading anything at compile time, because I’d like to have as many people using (and contributing!) to this library as possible, but keep in mind that our notions of what is convenient are very different.
Popular in Announcing
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex









