On Model-Checking Higher-Order Effectful Programs

Model-checking is one of the most powerful techniques for verifying systems and programs, which since the pioneering results by Knapik et al., Ong, and Kobayashi, is known to be applicable to functional programs with higher-order types against properties expressed by formulas of monadic second-order logic. What happens when the program in question, in addition to higher-order functions, also exhibits algebraic effects such as probabilistic choice or global store? The results in the literature range from those, mostly positive, about nondeterministic effects, to those about probabilistic effects, in the presence of which even mere reachability becomes undecidable. This work takes a fresh and general look at the problem, first of all showing that there is an elegant and natural way of viewing higher-order programs producing algebraic effects as ordinary higher-order recursion schemes. We then move on to consider effect handlers, showing that in their presence the model checking problem is bound to be undecidable in the general case, while it stays decidable when handlers have a simple syntactic form, still sufficient to capture so-called generic effects. Along the way, we hint at how a general specification language could look like, this way justifying some of the results in the literature, and deriving new ones.


INTRODUCTION
Verifying the correctness of programs endowed with higher-order functions is a very challenging problem, which can be addressed with various methodologies, from type systems [Davies and Pfenning 2000;Freeman and Pfenning 1991;Hughes et al. 1996] to program logics [Brady 2013;Jung et al. 2018], from symbolic execution [King 1976;Tobin-Hochstadt and Van Horn 2012] to verified compilation [Leroy 2009].An approach with some peculiarities is that of higher-order model checking (HOMC in the following), which consists in seeing the program at hand as a structure, then checking whether it renders a logical formula capturing the desired property true, namely whether it is a model of it.Saying it another way, HOMC can be seen as the application of the model checking paradigm [Clarke 1997;Clarke et al. 2018] to higher-order programs.One of the characteristics of this approach is that, contrary to most others, it is often both sound and complete.As a consequence, the kind of languages to which the methodology can be applied are very often not Turing-complete, the underlying verification problem being undecidable even for very simple logics.
The feasibility of higher-order model checking was scrutinized in the early 2000s, the objective being to extend classic results about model checking recursion schemes [Courcelle 1995] to higher-order generalizations of the latter.The results obtained were initially very interesting but partial [Knapik et al. 2001[Knapik et al. , 2002]], only concerning certain restricted forms of higher-order recursion schemes.The quest came to an end in 2006 with Ong's groundbreaking result [Ong 2006] on the decidability of the model checking problem for trees generated by general higher-order recursion schemes against formulas of MSO (or, equivalently, of formulas the -calculus or alternating parity tree automata [Emerson and Jutla 1991;Grädel et al. 2003]).This result was followed by many other ones [Broadbent et al. 2010;Carayol and Serre 2012;Hague et al. 2008;Kobayashi 2009;Kobayashi and Ong 2009;Salvati and Walukiewicz 2011;Walukiewicz 2016], whose goal was that of understanding the deep computational nature of the problem, at the same time generalizing the decidability result and building concrete verification tools [Broadbent and Kobayashi 2013;Kobayashi 2011;Neatherway et al. 2012;Ramsay et al. 2014], readers can refer to [Ong 2015] for an overview.
Among the various extensions of higher-order schemes considered in the literature, we should certainly mention extensions aimed at capturing higher-order recursion schemes subject to more permissive type disciplines than that of simple types, namely the one to which Ong's classic result applies.As an example, higher-order recursion schemes with recursive types have been recently considered [Kobayashi and Igarashi 2013].We should also mention some attempts at making the technique applicable to programs which are not pure, but which can produce, for example, nondeterministic and probabilistic effects.In the second case the decidability results scale back [Kobayashi et al. 2020], with undecidability showing up already at order three and for mere reachability properties.In the first case, instead results remain essentially unchanged [Tsukada and Kobayashi 2014].
The motivation from which this work originates is precisely that of understanding the deep reasons for the aforementioned discrepancy, at the same time giving a general account of the HOMC problem in presence of effects.In doing so, we will consider effects as being captured by algebraic operations [Plotkin and Power 2003], the latter producing some pre-defined effects or interpreted by way of effect handlers [Hillerström et al. 2017;Kammar et al. 2013;Plotkin and Pretnar 2009].In other words, we will consider well-established ways of capturing effects in higherorder -calculi.On the side of specifications, we analogously try not to consider ad-hoc formalisms, and look for conservative extensions of MSO in which the properties of interest can be captured in a unifying way.In particular, in the algebraic approach to effects, trees are always considered up to an equational theory, describing how the different algebraic operation should behave.In our approach, this means that it is particularly important to be able to define specifications that take into account this equational theory, and this is captured in our approach by the notion of an observation [Johann et al. 2010;Matache and Staton 2019;Simpson and Voorneveld 2019].
The contributions of this paper are threefold: • We first of all consider a finitary version of Simpson and Voorneveld's EPCF, a calculus with full recursion and algebraic effects, showing that the computation tree semantics of any EPCF computation  is precisely the one of a  -term  * obtained by CPS-translating .Since the  -calculus is well-known to be equiexpressive to higher-order recursion schemes [Nakamura et al. 2020], we obtain that computation trees generated by EPCF can be automatically checked against MSO specifications.This is in Section 4 and Section 5. • Then, we turn our attention to more general and more expressive specifications.Among the many proposals for logic for algebraic effects, we consider a variation on the one proposed by Simpson and Voorneveld, in which any effectful computations can be tested through the notion of an observation.We then prove that only certain notions of observations give rise to decidable model-checking problems, this way justifying some of the existing results in the literature, at the same time proving new ones.This is in Section 6. • Finally, we consider the impact of effect handlers to the HOMC problem.We show that handlers are indeed harmful to decidability, at least when general, well-established notions of handlers are considered, including shallow and deep handlers.We conclude by considering a rather restricted class of handlers which are sufficiently expressive to capture generic effects [Plotkin and Power 2003] but for which model checking remains decidable.This is in Section 7 and Section 8. l e t r = ref true ; ; l e t q = ref true ; ; l e t rec f g = l e t y = !r in l e t z = !q in i f (g y z ) then failwith ( " Failure " ) else begin r := (not z ) ; q := (not y ) ; f g ; end in f ( fun x y ↦ → x <> y) (a) An OCAML program P which never fails.
Fig. 1 All in all, the results above provide a rather clear picture about how far one can go in applying existing HOMC methodologies to effectful programs.The take-home messages are that in principle, such techniques can be reused, provided the underlying notion of observation does not give rise to too complex specifications, while handlers are potentially very dangerous, and should be used with great care.The technical core of the paper is in Section 3 to Section 8, while Section 2 serves as a gentle introduction to higher-order programming with effects and its verification.Related work is discussed in Section 11.

HIGHER-ORDER EFFECTFUL PROGRAMS, AND HOW TO MODEL-CHECK THEM
While the -calculus is the reference paradigmatic model for pure functional programming, a standard way of raising effects from within functional programs consists in invoking algebraic operations [Plotkin and Power 2003], each of them corresponding to a particular way of producing an observable effect.Even when the underlying programming language does not offer algebraic operations natively, many impure constructs can be interpreted this way.Consider, as an example, the OCAML program in Figure 1a, call it P, which manipulates two ground global variables r and q through a recursive higher-order function f.As can be easily realized, whenever the conditional is executed the two references r and q contain the same boolean value.As a consequence, the Failure exception is never raised, and the program can be considered safe.Could we automatically verify the latter by way of higher-order model checking?Let us try to see if this is possible.
We can assume Loc to be a type of locations inhabited by q and r only, and that the program can invoke any effect-raising operations from the following typed signature: Some standard abbreviations allow us to form the term in Figure 1b, call it  P , whose structure is very similar to the one of P. In doing so, we have adopted a syntax close to Simpson and Voorneveld's EPCF.Observe how OCAML's reference commands have become algebraic operations from Σ, and how Get has arity equal to two, accounting for the fact that the program can proceed depending on the value read from memory.By the way,  P closely corresponds to the way one would write P in languages like EFF [Pretnar 2015].
Set(r, False) Set(q, False) (a) Tree Generated by  P .But why do algebraic operation could help in the task of verifying the safety of P? Actually, the evaluation of programs which invoke algebraic operations naturally gives rise to a so-called effect tree.The effect tree produced in output by  P looks like one in Figure 2a.Through it, one can verify that Raise is never executed by exhaustively considering all branches of the tree, and verifying that those which are somehow coherent with the store operations do not end in a leaf labeled with Raise.Here, coherent branches are those that, e.g., when encountering Get() proceed left (respectively, right) depending on the last Set(, ) operation performed.This reasoning can indeed be encoded in an alternating parity tree automata, and thus by a MSO or -calculus formula [Emerson and Jutla 1991;Grädel et al. 2003].Let us briefly describe how this automaton can be constructed (here, by the way, we only need a top-down deterministic automata).Its set of states comprises all assignments of boolean values to the variables r and q, together with a special state OK.Intuitively, the latter captures incoherent states, and any action is accepted from there.Some example transitions are those in Figure 2b.The first two capture the expected behavior of Set and Get on stores, while the last three are there to model the fact that every action, including Raise is allowed in the state OK.Of course, the latter is not available in ordinary stores instead.
All in all, then, we indeed have a way to turn impure functional programs into terms of a calculus called EPCF which generate trees which are, at least superficially, amenable to model checking.There is still a missing link though: it could well be that EPCF is simply too expressive, and that the model-checking problem is undecidable.Fortunately, however, we can turn EPCF programs into  -terms, for which MSO model checking is indeed decidable [Nakamura et al. 2020].This is precisely what we prove in Section 5 below.
Summing up, HOMC indeed seems to be helpful when considered on effect trees, because the program we are interested in verifying, and arguably any term working with global references over finite domains can be turned into a term in the Y-calculus, while the property becomes an MSO formula.But how about other effects?Can we turn the construction above into something more general and systematic?While specific kinds of effects are considered in the literature [Kobayashi 2009;Kobayashi et al. 2020;Ong 2015], no general result is known.In fact, each effect comes equipped with its intended notion of observation [Dal Lago et al. 2017;Johann et al. 2010;Matache and Staton 2019;Simpson and Voorneveld 2019].Which ones of those are simple enough to guarantee that model checking useful properties stays decidable?
For the sake of convincing the reader that the questions above are not trivial, let us consider another example of an effectful higher-order program, borrowed from [Kobayashi et al. 2020] l e t F = return ( fix f ._ .
Flip ( b . case (b , return ( ) , l e t x = f ( ) in l e t y = f ( ) in f ( ) ) ) ) in F ( ) The algebraic operation Flip(.)should be understood as flipping a random coin, storing the result in , and continuing as .Intuitively, the program described above creates a procedure  that throws a coin.If the coin returns head, the procedure stop.Otherwise, the procedure is executed three times.This program generates an effect tree  0 where for every , the tree   can be defined as: Indeed, the sequential composition in the third argument of the case operator gives raise to a stack of continuations, meaning that a leaf return() is replaced by the tree computed by the continuation.
Here, the tree   should be understood as a call to the recursive function  with a continuation corresponding to  calls to this function.This behavior is similar to the one of a random walk, but the program described above does not use any infinite type, which is essential for Higher-Order Model Checking.In fact, this tree can be computed by a term of the Y-calculus, using a CPS translation, see Example 2.10 from [Kobayashi et al. 2020]: ( (, ,  .Flip ( ) ( ( ( )) ))) ( .)return Here, the use of higher-order functions is important, since the function  takes as a first input a continuation, and this continuation determines the number of calls to the procedure that still need to be executed.For such a program, it is natural to wonder whether return is called or not (if the program terminates), but given the presence of (probabilistic) nondeterminism, there are various ways in which this can be spelled out.Do we mean that the program must (or may) reach return?Or do we rather mean that the program reach return with probability 1?The latter question seems the most appropriate, given that probabilistic choice is captured by the subdistribution monad D(•) and that D(return) is just a real number between 0 and 1, i.e., the probability of not diverging.
If this is the case, however, recent results by Kobayashi, Dal Lago, and Grellois [Kobayashi et al. 2020] show that HOMC is not decidable in general, so the construction of an MSO formula (or an automaton) like the one above is simply not possible.May or must termination, instead, can easily be captured: may termination consists in exploring the tree and finding at least one return, while must termination is satisfied if the tree has no infinite branch, and all leaves are return leaves, which is a property that can be encoded as a parity condition of an alternating parity automaton.
To sum up, we can see effectful higher-order programs as effect trees, computed by a term of the Y-calculus (or equivalently, higher-order recursion schemes) for which we know that MSO model-checking is decidable.This works particularly well for some effects, for example global store with finite domains, since the properties of interest can be expressed as an MSO formula.However, for some more involved effects, such as probabilistic choice, some important properties are not expressible in MSO.The aim of this work is to take a general look at this problem, and understand the deep reason behind this discrepancy.In Higher-Order Model Checking, models are traditionally taken to be infinite trees produced by so-called Higher-Order Recursion Schemes (HORS in the following).In this work, we rather consider Böhm trees generated by ground type terms in the  -calculus with first-order constants, which is well known to express the same class of trees as HORSs [Nakamura et al. 2020].This section is devoted to presenting some preliminaries about the calculus and the model checking problem for it.

Γ, 𝑥
Formally, the  -calculus can be seen as the simply-typed -calculus extended with full recursion and with first-order function symbols: The signature Σ is a set of first-order constants, such that each  ∈ Σ comes equipped with an arity ar( ) ≥ 0 capturing the fact that the type of  is  →  →  ar( ) → , often abbreviated as  ar( ) → .
By an abuse of notation, we usually write  : ar( ) to specify the arity of  in a given signature when this does not cause ambiguity.Typing rules are standard, and can be found in Figure 3.
We see the  -calculus as a tool to generate infinite trees.In order to precisely define the tree generated by a typable term, one has to define a form of dynamic semantics, which we here take as weak head reduction: we never reduce the argument of any application, and we never evaluate terms in the scope of -abstractions.Rules are again standard, and can be found in Figure 4.It is relatively easy to see that typed closed terms in weak head normal form, i.e. typed terms with no free variables that cannot be further reduced are precisely the (typable) terms generated by the following grammar: Indeed, a -abstraction is in normal form, constants are in normal form, and   is in normal form if and only if  is in normal form and  is not a -abstraction.As a consequence, we can easily realize that terms of ground types in weak head normal form have the shape   1  2 • • •  ar( ) , where each   is itself a term of ground type.This naturally suggests a potentially infinite process turning any such term of ground type into a tree with root  and ar( ) subtrees obtained by evaluating  1 • • •  ar( ) , respectively.This can be made formal as follows: Definition 1 (Infinite Trees).The set of (potentially infinite) trees generated by a signature Σ, denoted Tree Σ , is coinductively defined by the grammar: where  : ar( ) ∈ Σ.
The constant ⊥ represents a non-terminating non-productive computation that does not generate any function symbol.As an example, if Σ is the signature { : 2,  : 1,  : 0}, one can form the non-regular infinite tree (, ( (), ( ( ()), • • • )).We are now ready to define how any closed ground Y-term generates such an infinite tree: Definition 2 (Böhm Trees of Closed Ground Terms).Given a closed term  of type , i.e. we have • ⊢  : , the Böhm tree of , denoted  (), is defined by way of the following essentially infinitary process.Starting from , we apply → ad infinitum.This can have two possible outcomes: •  can be reduced infinitely, and in this case  () is simply ⊥ •  can be reduced to a term  in weak head normal form The aforementioned process is infinitary in two different ways: the evaluation of  can diverge, and  () can be infinite.The process above is well-defined only for closed terms of ground type and is thus less general than the one generating Böhm Trees for arbitrary terms of the  -calculus [Clairambault and Murawski 2013] (there, in particular, one has to deal with -abstractions).

Expressing Properties As Alternating Parity Tree Automata
Now that we have defined the class of models for higher-order model checking, we can define the specification language.The gold standard in model-checking consists in properties expressed by formulas of monadic second order logic [Knapik et al. 2001] (MSO for short).However, several equiexpressive specification languages have been defined in the literature, notably -calculus [Walukiewicz 1993] and alternating parity tree automata (APT for short) [Emerson and Jutla 1991;Grädel et al. 2003].The latter is commonly used in the HOMC literature [Kobayashi and Ong 2009;Ong 2006Ong , 2015]], and as examples we will define through the paper have a simple representation as automata, contrary to MSO formulas, we also chose this specification language.
Given a set  of variables, we define the positive Boolean formulas over  , denoted  + ( ), with the grammar: with  ∈  .A subset  ⊆  satisfies a formula , denoted  ⊨  if and only if the formula  in which all elements  ∈  are replaced by tt and all elements  ∉  are replaced by ff is semantically true.We can now give the formal definition of an APT automaton: Definition 3 (Alternating Parity Tree Automaton).An alternating parity tree automaton is a tuple A = (Σ, , ,   , Ω) where • Σ is a signature, defining tree constructors with their arity.
•  is a finite set of states, with   ∈  the initial state.
To define the acceptance condition of an automaton, we need to introduce the notion of positions in a tree.Definition 4. Let Σ be a set of tree constructors with maximal arity .The set of all position of a tree , denoted (), is a set of words on the alphabet Γ = {0, • • • , } defined by: where • is the concatenation operator for words.
For a tree  and a position  ∈ () we define  () ∈ Σ ∪ {⊥} by: essentially giving the node of the tree  at position .
A run-tree of an automaton A over a tree in Tree Σ is a tree with constructors in () × .This run-tree must satisfy the following two constraints: • The root is (,   ), representing the root of  in the state   .
• For any node (, ) of the run-tree, there is a set  ⊆ {0, . . ., ar( ()) − 1} which satisfies the transition  (,  ()).And, for each (,  ′ ) ∈ , one of the child of (, ) in the run-tree is ( • ,  ′ ).Finally, we say that an infinite branch (,   ) • • • (  ,   ) • • • of the run tree satisfies the parity condition if the largest priority that occurs infinitely often in Ω(  ) • • • Ω(  ) • • • is even.A run-tree is accepting if every infinite path in it satisfies the parity condition.And of course, a tree  ∈ Tree Σ is accepted by an automaton A if there exists an accepting run-tree for .
Example 2. We define an automaton A = ({ : 2,  : 1,  : 0}, { 0 ,  1 }, ,  0 , Ω) such that A accepts a tree  if and only if for every path in ,  occurs eventually after  occurs.We pose: • For all  ∈ { 0 ,  1 } : (, ) = tt Meaning intuitively that a node  propagates the state to both children, seeing the letter  changes the state to  1 and a leaf  is always accepted.
By definition, if  1 appears infinitely often in a branch of the run tree, and  0 does not, then the largest priority is 1 in this branch, and the tree does not satisfy the parity condition.Thus, with this automata, to accept a tree , it must have only finite branches (that terminates with the only available leaf ) or infinite branches in which  does not occur (in order to stay in the state  0 ).This corresponds indeed to the property that for every path in ,  occurs eventually after  occurs".
We can now define the model-checking problem and give the theorem expressed in [Ong 2006].
Definition 5 (MSO Model-Checking for Böhm Trees (or APT Acceptance Problem)).Given a closed Y-term  of with ⊢  :  and an APT A, is  () accepted by A ?
The groundbreaking result of higher-order model checking [Ong 2006] can then be summed up with the following theorem: Theorem 1 (Decidability [Ong 2006[Ong , 2015]]).MSO model-checking on Böhm Trees of closed ground terms of the Y-calculus is decidable.
The results obtained in [Ong 2006] give the precise complexity of this problem, which is exponential depending on the order of the Y-term (or, equivalently, the order of the higher-order recursion scheme [Nakamura et al. 2020]).The aim of our work is on decidability, so we will not take orders into account, but we plan to study complexity issues in the future, given that finer complexity analysis for similar problems is already available [Nakamura et al. 2020;Tsukada and Kobayashi 2014].
Thanks to this theorem, to show that MSO model-checking is decidable for a class of infinite trees, it is sufficient to show that this class of trees can be computed as Böhm trees generated from Y-terms, and this can be done via program transformation to the Y-calculus, as we will see in many occasions starting from the next section.

HIGHER-ORDER PROGRAMS WITH EFFECTS
In this section, we introduce a calculus with algebraic operations and fixpoints, called EPCF.This language can be seen as a fine-grained call-by-value variation on Plotkin's PCF endowed with effect-triggering operations, as described in [Simpson and Voorneveld 2019].We consider a finitary version of this language which differs from Simpson and Voorneveld's one only in minor ways.
EPCF is built around two syntactic categories, namely values, which denote data or functions, and computations, which are instead programs which potentially produce effects when evaluated.Terms and types for EPCF are as follows: We suppose given a set of finite ground types, ranged over by B. Examples include the unit type and the type of booleans.We similarly assume a set V of constant values each of a ground type, ranged over by , e.g. a unique constructor () ∈ V with its associated type Unit.We also assume finite enumeration types k to be present, each of them inhabited by the values 0, 1, . . .,  − 1.We introduce enumeration types, which support pattern matching, just to keep them distinct from finite base types.
Algebraic operations have the shape  ( ;  .)where  is said to be a parameter, and  represents the continuation of the computation.The variable  is bound in  and represents the choice made by the algebraic operation.We suppose given a signature set Σ of symbols for algebraic operations, where each  ∈ Σ is given an arity B ⇝ , meaning that  is an algebraic operation of arity  depending on a parameter of type B. In an operation of arity , the type of  in the continuation  is k.In particular, defining  different continuations  1 , . . .,   depending on the possible values of  can be done by way of the computation  ( ;  .case(, 1 , . . .,   )).As common in call-by-value calculi, the fixpoint operator fix  . is a value and can only be attributed a function type without limiting the expressive power of the language.Finally, the shape of computations is restricted in that the only way to compose computations is to use the let constructor.For example, if we want to apply the result of the computation  to the function computed by , we need to define a computation As can be seen from this example, the order of evaluation is imposed by sequencing, a property which turns out to be crucial in presence of effects, as we will see when looking at the semantics.
As for the type system of EPCF, which is pretty natural and standard, we give some of the rules in Figure 5.The typing rule for algebraic operations stipulates that an effect does not change the type of the whole computation, which is however arbitrary.Intuitively this is because calling an algebraic operation induces an effect , depending on the parameter  , and the computation will continue as  after throwing this effect.
We now introduce the dynamic semantics of EPCF using the reduction relation in Figure 6.The cornerstone of this relation is the last rule, allowing to algebraic operations to "percolate out" of non-trivial evaluation contexts.Intuitively, this rule means that if we have a computation of the shape let  =  in , and  is a computation that throws an effect , then this effect is immediately visible from the outside, its continuation now incorporating the let expression at hand.If we go back to the aforementioned example describing the composition of computations, namely the pair of terms we can see that in the first case, the effects thrown in  appears first, and then the one in , and conversely for the other case.
Example 3. Let us consider a computation which flips a fair coin, then returning the the exclusive or of the obtained value and itself (which is thus always going to be true).All this makes use of an algebraic operation Flip having arity unit ⇝ 2 and of a binary function ⊕ computing the exclusive or, and which can anyway be easily written as a term itself: In a call-by-value setting, this program should flip a coin, to obtain 0 or 1 thus always returning 0. Indeed, the dynamic semantics allows the Flip operation to be visible from the outside: There are two observations one needs to make now.The first is that the continuation let  =  in  ⊕  returns 0 for all values of , as expected, but since it occurs as the argument of Flip it cannot be further reduced, and its evaluation must be somehow be captured differently.The second one is that in presence of effects, confluence is lost.If, indeed, we pass Flip(•; .) unevaluated to the right-hand side, in a call-by-name fashion, we land on a totally different term, namely one which flips two coins, possibly returning 1: The example above implicitly tells us that, similarly to what happens in the  -calculus, the terms of EPCF generate infinite trees that we commonly call effect trees, but that the dynamic semantics does not by itself take care of the unfolding: indeed, typed closed computations in normal form in EPCF are either of the shape return( ), which is expected, or  ( ;  .),meaning that evaluation apparently stops as soon as an algebraic operation is encountered.The effect tree of a computation is what we are looking for as is defined as follows: Definition 6 (Effect Trees for EPCF Computations).For an algebraic effect signature Σ and a type  , let us define the signature Σ  as follows: Note that Σ  is infinite in the general case, but that it is guaranteed to be finite when V is finite and  is a ground (and thus finite) type.The effect tree of a typed computation ⊢  :  , denoted  (), is a tree in Tree Σ  defined by the following essentially infinite process.First of all, reduce  ad infinitum, and there are three possible outcomes: • If  can be reduced infinitely without reaching a normal form, then This definition of effect trees differs a bit from the one given in the literature by the use of a special branch for parameters.Formally, standard effect trees [Matache and Staton 2019;Simpson and Voorneveld 2019] would be defined with the signature: } with the expected definition when seeing an effect  (;  .)).Our definition is not harmful compared to this usual notion of effect trees in which operations are indexed by their parameters because we are in a finite case, where the parameters can only be a constant value for some finite type B. In particular, any MSO formula on a standard tree with indexed effects for can be transferred to an equivalent formula on effect trees with branching for parameters, see Section 6, Proposition 3 for more details.Anyway, one can remark that this definition is similar to the notion of Böhm trees we defined before.And indeed, we show that we can relate the notion of effect trees in EPCF with the notion of Böhm trees in  -calculus.Formally, we need to use a CPS translation similar to the one described in [Matache 2018;Matache and Staton 2019].
Example 4. As an example, we can compute a tree with similarities with the one of Example 1 in the Y-calculus.We take Σ = {  : Unit ⇝ 1,   : Unit ⇝ 2}, and the EPCF computation  ≜ fix  .(. ((), .case(; ,  (),  (. ((), _. ())))) For the sake of clarity, let us ignore all the parameters for algebraic effect, which do not bring anything interesting here because the parameter is always ().Then, we have: And if we continue the computation, we can see that the effect tree  of  (.return(())) is  ≜   (return(()),   (  (return(())), • • • ) Note that because of the call-by-value paradigm, we need to use thunk functions, something which is not needed in the Y-calculus.As expected, in the translation from EPCF to Y-calculus, we have to take into account of all this along the CPS translation.

FROM EPCF TO THE 𝜆𝑌 -CALCULUS
It is now time to turn the informal arguments we made at the end of the last section formal.Indeed, translating any EPCF computation into a term of the  -calculus is indeed possible, as we are going to show.
The fact that EPCF ground values, arities, and parameters are all taken from finite types implies that for every algebraic effect signature Σ, the signature Σ  defined as This enables the defining of a transformation from EPCF to the  -calculus with constants in Σ  , which we first give on types: Definition 7 (CPS Type Transform).For the sake of conciseness, we use ¬ to denote the type  → .We pose Remark that this translation can be done because of our choice of effect trees with a special branch for parameters, indeed it is not possible at runtime to know exactly the value of the parameter  of an algebraic operation, and with our definition of effect trees, we can postpone the identification of  * by just passing it as an argument.This translation is well-typed, as shown by the following lemma, whose proof is straightforward: Then, we would like to show that this translation is a simulation.But, in order to do that, we need to take into account that there are some reductions that need to be done outside weak head reduction in Y, but those reductions are actually administrative [de Groote 1994; Hillerström et al. 2017;Plotkin 1975].Formally, we introduce a new reduction relation → adm by way of the following rules: Intuitively, an administrative reduction here consists in applying a -rule, and possibly in the argument of an application (which not possible in head reduction).We take this small set of rules as it is exactly what we need to have the simulation, and it should be intuitive that applying those rules has no consequence in the construction of a Böhm tree of a Y-term.Remark that we could have defined the transformation of a  using ( .* ) 0 * instead of directly  * [ := 0 * ] if we wanted to avoid using substitution in the encoding.But this would induce more administrative reductions.
Lemma 2. If  →  then for all continuation ,  *  → +  such that  *  → ≤2 adm .Proof.The proof can be done by induction on  → .All cases are straightforward and do not need administrative reduction except for the rule let  =  ( ; .) in  →  ( ; .let  =  in ), in which we do need administrative reductions: This means in particular that if a computation  never reaches a normal form, then  *  does not too.Indeed, the only case when we use the administrative reduction is the rule for a let with an algebraic operation, and if this rule can be used in a computation, then the computation will have a normal form, beginning with this effect.Then, by comparing the normal form of  and  * , we obtain the following theorem.
Theorem 2. For any ⊢  :  , for all ⊢  : Thus,  ( * ) simulates perfectly the effects of , but for the return values, instead of a leaf we have a tree representing the effect applied to this value by the continuation.As a consequence, using standard results of Higher-Order Model Checking, we have: Corollary 1.For any ⊢  :  and ⊢  :  * → , MSO model-checking on  () [return( ) ←  (  * )] is decidable.
In particular, if the continuation always return a new constant  for example, then  ( * ) is equal to  () with all return leaves replaced by .Also, if the type  is a ground type, then  can be taken as the identity, and we have exactly decidability of MSO model-checking for  ().In general, the set of all possible values in the form return( ) for a type  is infinite, as they could be -abstractions for example, but the previous theorem gives us a degree of freedom as we can always chose the continuation that we want in order to distinguish some of those values.

SPECIFICATIONS THROUGH OBSERVATIONS
In the previous section, we proved that EPCF computations can be translated into equivalent (in the sense of the tree they generate)  -terms.But how can we actually exploit this result for the sake of verifying interesting properties about the effect produced by EPCF programs?Of course, we can automatically verify whether any MSO formula has (the effect tree generated by) any EPCF term as its model.But, is it the end of the story?
As already, mentioned, there are many proposals in the literature [Matache and Staton 2019;Plotkin and Pretnar 2008;Simpson and Voorneveld 2019] about logics specifically designed for the sake of expressing properties of interest for effectful (possibly higher-order) programs.In this paper we will somehow follow the path recently hinted at by Simpson and Voorneveld [Simpson and Voorneveld 2019].
Effect trees as we have used them so far are not subject to any equational theory, the latter often being associated to effects and justified by a monadic interpretation [Moggi 1988].As an example, the following equations hold in the state monad, itself a way to interpreting the first example from Section 2: As pointed out in [Simpson and Voorneveld 2019], we can capture equations between effect trees agnostically, without the need of explicitly referring to monads, through the notion of an observation, namely a set of properties of effect trees: Definition 9 (Observations [Simpson and Voorneveld 2019]).For a given set of algebraic operations Σ, we define a set of observations as a set, denoted O, such that each  ∈ O is a subset of the effect trees of type unit.
On top of a set of observations O, Simpson and Voorneveld define a logic endowed with a modal operator of the form , where  is an element of O and  is a formula.An effect tree satisfies  if it is among the trees in  and all its leaves satisfy .This way, the logic can observe the tree at hand through the lenses of the observations in O, implicitly accounting for equations: it is necessary to define O such that all its elements are invariant by the desired set of equations.The resulting logic is quite powerful, being infinitary in nature, and turns out to precisely capture observational equivalence [Dal Lago et al. 2017] for (a calculus closely related to) EPCF.Noticeably, most notions of effects, including nondeterministic, probabilistic and state effects can be captured this way [Simpson and Voorneveld 2019] Unfortunately, however, the aforementioned logic is strictly more powerful than MSO or the -calculus.This happens for two distinct reasons: on the one hand, it is essentially infinitary in nature, this way accounting for the fact that the target calculus has countable types.On the other hand, the underlying set of observations O can be arbitrarily complex, thus injecting a huge discriminating power into the logic, something that MSO cannot do natively.
Motivated by all that, we address the following question in the rest of this section: can we express useful properties of effect trees as observations from within MSO itself?In other words, is it the case that for a given notion of effect, all the elements of O are captured by MSO formulas?This way, we could at least be sure, given the results from Section 5, that model-checking is decidable relative to O.This is precisely what motivates the following definition of an extension of MSO (or -calculus) with observation.
Definition 10 (Observation Predicate).Suppose given a signature Σ for effect trees, containing a set of algebraic operations, and a set of other possible constructors (that could come, e.g., from parameter values, or special constants of the Y-calculus).Fix a set of observations O. Let  ∈ Tree Σ , and  a subset of all the possible constructors of arity 0. We define  [∈ ] as the tree obtained from  by replacing the leaves in  with return(()) and all the other leaves with ⊥.We next define, for each  ∈ O and each , a formula   , called an observation predicate, such that for any effect tree t, we have Note that we have indeed that  [∈ ] is an effect tree of type Unit, thus it can be an element of .Write MSO O for the conservative extension of MSO (on infinite trees) obtained by enriching the class of formulas with observation predicates.
This notion of observation predicate having access to a set  is, thanks to finiteness, equiexpressive to the formulas  defined in [Simpson and Voorneveld 2019], in which  is supposed to be a formula that separates the correct leaves from the incorrect ones in an infinite set of leaves.Here, because of finiteness, we have a finite number of such separations, thus we can take a set  instead of a formula .
MSO O can thus be seen as a reasonable logic for writing down specifications about higher-order effectful programs.But is the model checking problem emerging out of it decidable?There are at least two possible answers to this question: • Either the observation predicates from O are definable in MSO itself, in this case MSO O is equiexpressive with MSO, and the model checking problem remains decidable.• Or observation predicates are not MSO-definable, and we cannot conclude, with the concrete risk of landing into an undecidable verification problem.We will now see some cases of effects and observations, each of them corresponding to one of the two cases above.Let us start from an example of effect which is hard to capture.
Example 5 (Probabilistic Observations).The typical counter-example of effects to MSOdefinability of observation predicates comes from probabilistic effects.The set of observations for probabilistic choice, with a unique operation Flip : unit ⇝ 2, is given by Essentially, an observation predicate  > means that the probability that the program terminates is greater than .This is not a property which can be defined in MSO on effect trees, and this happens for very good reasons, since we know that higher-order model checking for probabilistic HORSs is in general undecidable [Kobayashi et al. 2020]).
Before going into positive examples, we need to solve a small discrepancy between the literature on algebraic effects, especially with the standard definition of effect trees [Simpson and Voorneveld 2019].Indeed, as explained in Section 4, the Definition 6 that introduces effect trees in our work is slightly different from the standard notion of effect tree, which was essential for the translation of Section 5.However, in practice we would like to work directly with standard effect trees, as we did in Section 2, since they are simpler and corresponds to the literature.Concretely, we can show that our definition is not harmful for the question of MSO model-checking, since we can transfer any MSO properties (or APT) on standard effect trees to properties on effect trees with a special branch for parameters.Formally, we have: Definition 11 (Adding a branch for parameters).For any tree  defined with the signature where  is a set of possible other constructors (coming from a chosen continuation in the Y-calculus according to Corollary 1 for example), we define the tree   with the signature . This translation corresponds exactly to the translation of a standard effect tree to the one we introduced in Definition 6.
We can show that any formula (or APT) that is definable in standard effect tree in Tree Σ ′ is definable for the effect trees in Tree Σ of Definition 6. Proposition 3 (Effect Trees and APT).For any APT A that recognizes standard effect trees in Tree Σ ′ , there exists an APT A  that recognizes the set {  ∈ Tree Σ |  is accepted by A} Proof.The proof consists in adding a special state   for each variable  ∈ V and define transition as a big or depending on the value of the parameter.Formally, we define A  = (Σ,   ,   ,   , Ω  ) such that: • For the transition function we have: - where  [+1] is the formula  in which all couples (,  ) are replaced by ( + 1,  ).-For all  ∈ V,   (  , ) = tt and   (  ,  ) = ff for all  ≠ .
Thus, intuitively, when we reach a node , only one of all the (0,   ) will be satisfied, the one corresponding to the actual parameter  of .And then, the only satisfiable formula in this big or would be  (,   ) [+1] ∧ (0,   ), meaning that the transition is logically equivalent to  (,   ) [+1].Thus, we can see that the new transition   (, ) is equivalent to the actual formula  (,   ) [+1] where  is the parameter of  in the tree   , and from this it is straightforward to see that A  recognizes the set {  |  is accepted by A}.

□
Let us now turn to examples of observations which are indeed MSO-definable.
Example 6 (Non-determinism).Let us consider the unique effect { : Unit ⇝ 2}.The notion of observations for non-determinism is given by the two modalities ⋄ and □, where ⋄  is the set of effect trees that has at least one finite branch with a leaf in , and □  is the set of effect trees that has only finite branches, and all of them has a leaf in .
Those observations are definable by an APT.For the sake of the example, we explain informally the encoding of □  .We take a unique state   .To impose finite branching everywhere, it is sufficient to pose Ω(  ) = 1, because with this no infinite branch satisfies the parity condition.Then, We can just explore through the tree, with the transition  (  ,  ) = (0,   ) ∧ (1,   ).And finally, we can separate leaves with  (  , ) = tt if  ∈  and  (  , ) = ff if  ∉  for all leaves .
Example 7 (Finite State Monad).Going back to the examples used in Section 2, we start with the state monad, where the effects are Get and Set, with finite domains.As expressed in [Simpson and Voorneveld 2019], the set of observations for this effect is defined by O = { ↦ → | ,  are memory states} with the intuitive semantics that  ↦ → contains all effect trees mapping the state  to the state  (this computation is deterministic because the initial state is fixed, so we always know which branch should be taken on a Get).It is easy to see that the observations predicates coming from this set are definable in an APT, because an observation predicate ( ↦ → )  can be represented by an automaton starting from the state , that accepts only when the unique path leads to a leaf  ∈  in the state , similarly to the automata defined in Section 2. Thus, as expected, we can deduce that many interesting properties can be decided on programs using a finite state monad, since the observations are MSO-definable.

A CALCULUS OF EFFECTS AND HANDLERS
An algebraic operation in a language such as EPCF comes out uninterpreted: a computation produces an effect tree, in which all algebraic operations are tree constructors, and no meaning is given to operations in this tree.The notion of observation can indeed be seen as a way to at least identify trees which should be considered the same, implicitly capturing equations between them.Observations, however, attribute a static meaning to programs, and the programmer has no control over them.There is however yet another way of attributing meaning to effects in which the programmer has direct control on what happens.This can be seen as an abstraction on the notion of an exception handler, but also of, e.g., the HASKELL's monad construction.We are referring of course to the so-called effect handlers [Kammar et al. 2013].
In this section, we present a calculus obtained by endowing EPCF with effect handlers, called HEPCF, whose definition closely follows the literature on the subject [Hillerström et al. 2017;Kammar et al. 2013].This is necessary to even understand what doing model checking actually The language HEPCF differs from EPCF both at the level of terms and at the level of types.The former are extended with the handle clause, in which a computation is evaluated in a protected environment, in such a way as to be able to capture (and handle) the effects it raises.The latter are enriched with annotations keeping track of which ones among the operations are visible.The syntax of HEPCF is defined as follows: We add a new information in the type  →   , denoting the set  of available algebraic operation the function at hand can possibly perform once applied to an argument of type  .An handler consists of a return clause return() ↦ →   and one clause   (;  ) ↦ →   for each handled operation   in some effect set .The intuition behind such an handler is that it takes a computation using the effects in {  | 1 ≤  ≤ }, and interprets each call to the algebraic operation   as the computation   in which the continuation is passed as a variable  .The return computation   is called when the initial computation returns a value.
Selected rules of the type system are in Figure 7.A judgment attributing a type to a computation  now has the shape Γ ⊢   :  , meaning that  has the type  in the effect context .As the reader can see in the rule for a call to an algebraic operation, a computation of type Γ ⊢   :  can only give rise to the operations in .As expected, a handler takes a computation  of some type  that uses the effects in  ′ and transforms this computation into a computation of type  using the effects in .This operation is very similar in principles to an application, that is why we describe the type of an handler with an arrow type.In details, the type   ′ ⇒   means that all the computations defined in the typed handler should use only the effect in , and the handled operations are exactly those in  ′ .In such a context, a continuation for an algebraic operation is given as the variable  of type  →   , meaning that this continuation is, as expected, of type  Finally, the dynamic semantics of HEPCF is described in Figure 8 and Figure 6.As expected, handling a returned value consists in calling the return clause, and for the case of an algebraic operation, it is important to note that the handler acts also on the continuation.As for the definition of an effect tree, we can see that as before, a closed typed computation ⊢   :  in normal form is either an algebraic operation in  or return( ), thus we can define effect trees similarly to Definition 6, with the signature  instead of Σ.Note that we can see the typing ⊢  :   ′ ⇒   as defining a tree operation from effect trees on  ′ of type  to effect trees on  of type  .In practice, this transformation takes a tree defined by ⊢  ′  :  and gives a tree defined by ⊢  with  handle  :  .We will see in the next section that, unfortunately, the tree transformations that can be expressed by handlers form a very large class, definitely too broad for our purposes.

MODEL CHECKING HANDLED EFFECTS: POSITIVE AND NEGATIVE RESULTS
In the previous section, we introduced effect handlers, a very powerful and elegant linguistic construction by which the interpretation of algebraic operations can somehow be delegated to the programmer.In this section that this notion of handlers is too expressive for our purposes, meaning that MSO model-checking of effect trees produced by terms of HEPCF is not decidable.We will also show that when handlers have a restricted but non-trivial form model checking becomes decidable.

Undecidability Through the Halting's Problem
The proof of undecidability is structured around the encoding of Plotkin's PCF into HEPCF.The encoding is not trivial, because PCF is taken in its full generality (thus including a type of natural numbers) while HEPCF only has finite base types.Since the halting problem for PCF is well-known to be undecidable, and can be written down as an MSO formula, undecidability of the HOMC problem for HEPCF easily follows.Formally, we introduce standard PCF as the following language: Definition 12 (Plotkin's PCF).The PCF language (in fine-grained call-by-value) is defined by the following grammar: This is a well-known language for which the halting problem is not decidable: encoding all partial recursive functions is an easy exercise.We omit the definition of the static and dynamic semantics, which are anyway natural and very standard.
In order to prove the undecidability of HEPCF we show a translation from PCF to HEPCF.Again, note that the crucial difference between the two languages is that PCF has access to an infinite type Nat for natural numbers, with a pattern matching constructor, whereas in HEPCF there are handlers and algebraic effects but all base types must be finite.But how could we encode the infinite type of natural numbers using finite types, effects and handlers?In order to do this, we introduce the effect  = { : Unit ⇝ 1}.For the sake of simplicity, we will ignore the parameter for this operation, of type Unit.Intuitively, we will represent a natural number  by a computation in which this operation  is called exactly  times, and then the computation halts by returning the only inhabitant of the type Unit.But, because a natural number  is supposed to be a value, we will use a thunk function   along the encoding, and each time we need to inspect the value of a natural number , we can call   () to produce the effect tree corresponding to the tree representation of this number .
Definition 13 (Translation from PCF to HEPCF).For any type  of HEPCF we define the zero of this type, denoted Z  as follows: Intuitively, the zero of a type is just a particular closed value of this type chosen arbitrarily, in which no effects are used.For the sake of clarity, we introduce a notation for several -abstractions and several applications in a row: for which we obtain, as expected, that ( , .) where  (, ) is defined, when  and  have type  , by: Let us now give some hints about the translation.As expected, zero is just the function that return () when called, and a successor is a call to the effect .In order to do the pattern matching for a natural number  given as a thunk function ⟦ ⟧, we produce the effect tree representing , by calling ⟦ ⟧(), and we handle it.This handler produces a pair of thunk computations.The first computation is just a copy of the initial computation ⟦ ⟧().This is rather obvious for the return clause, and for the case of , the value  corresponds intuitively to just taking the first computation in  , thus a copy of the predecessor, and in the returned pair of computation, the first pair is  applied to this , which gives the successor of the predecessor, i.e. a copy of the current number.The second computation simulates the initial pattern matching.For the case of a zero, the second computation is just , and for the case of a successor (an operation ), the second computation takes the copy of the predecessor given by , and then returns the computation  that uses this predecessor.
The proof that this translation is correct is inspired by the proof of the encoding of shallow handlers using deep handlers, described in [Hillerström and Lindley 2018].Intuitively, this is because a shallow handler is a handler that only handles the root of an effect tree, which is exactly what a pattern matching here is supposed to do.First, we can show that this translation is well-typed: Lemma 3. If Γ ⊢  :  then ⟦Γ⟧ ⊢ ⟦ ⟧ : ⟦ ⟧ and if Γ ⊢  :  then ⟦Γ⟧ ⊢  ⟦⟧ : ⟦ ⟧ Proof.The proof is direct except for the case construct, where we need to prove this intermediate result: If Γ ⊢  :  , Γ,  : Nat ⊢  :  , ⟦Γ⟧ ⊢ ⟦⟧ : ⟦ ⟧ and ⟦Γ⟧,  : ⟦Nat⟧ ⊢ ⟦⟧ : ⟦ ⟧, then With this type in mind, the proof that  (, ) respects this type is straightforward, and we can conclude that the translation for case is well-typed.□ We want to prove that the translation is a correct simulation.In order to do this, we need to introduce formally an approximation of a term up to some administrative context, as in [Hillerström and Lindley 2018], because in this translation we encode integers as functions, and we will sometimes have a term that is overly complex because all the computation is hidden behind a , but this term will behave exactly like another simple term corresponding to the encoding of an integer.This is exactly what happens here; the variable  in the handler is essentially a copy of the predecessor, but syntactically  is described as a complex computation.Definition 14 (Administrative Contexts (See [Hillerström and Lindley 2018],Definition 6)).Evaluation contexts are defined with the following grammar: They corresponds exactly to context in which a computation can be reduced.An evaluation context E for a computation is called administrative, denoted adm(E) if and only if: • For all values  , E [return( )] → * return( ) • For all operations , E [ ( ;  .)]→ *  ( ;  .)with  → * E [].
Remark that the composition of two administrative contexts is also an administrative context.
Definition 15 (Approximation up to Administrative Reductions (See [Hillerström and Lindley 2018],Definition 7)).We define ≥ for terms as the closure under any context of the following rule: Intuitively, we have  ≥  when  could be reduced to  using strong rules of reduction (reducing under any context) without changing normal forms (of the form return or ).
We can then prove the following lemma on this approximation, showing that the properties of administrative context are valid for approximations.Lemma 4. If  ′ ≥  then: Proof.We proceed by induction on  ′ ≥ .All the contextual rules are straightforward because  ′ has directly the desired shape in this case.
• If  ′ ≡ E [ ′′ ] with  ′′ ≥ , then we have, by induction hypothesis, if  ≡ return( ) then then this is direct by induction hypothesis.

□
We can then show that this approximation is a simulation.Lemma 5.If  ′ ≥  and  →  then there exists  ′ such that  ′ ≥  and  ′ → +  ′ Proof.We proceed by induction on  ′ ≥ .
• If  ′ is E ( ′′ ) with  ′′ ≥  and (E).Then, by induction hypothesis, there exists  ′′ with  ′′ ≥  and  ′′ → +  ′′ .We pose  ′ ≜ E ( ′′ ).Because E is an evaluation context, we have E ( ′′ ) → + E ( ′′ ), and because E is administrative, we have E ( ′′ ) ≥ .This concludes this case.• If  ′ →  ′′ with  ′′ ≥ .Then by induction hypothesis, there exists  ′′ such that  ′′ → +  ′′ with  ′′ ≥ , and we can pose -The case  = fix . is similar to the previous one, we take the obvious reduction and it concludes.•  ′ cannot be of the shape return( ) or  ( ;  .)because it would mean that  is in normal form, and thus cannot be reduced.
There are three cases to consider depending on the reduction  → .
Proof.For this proof, we will consider that all values of type unit are equal.We proceed by induction on  → .All cases are straightforward except for the case, that we detail here.
• Consider the reduction case(0; 0 ↦ → ; succ() ↦ → ) → .We have This concludes this case.We can show that the following context: -We have: with  (, ) handle  (.) so let us call this computation  ′ for the sake of clarity.We have: with  (, ) handle  (.) → + let  = return( .′ ) in return(.( .(. ())) ( .⟦⟧)) By going back to the previous computation in the whole context, we obtain: And let  =  in return(()) with  of type unit is -equivalent to .This concludes this case.-For the case of return, we have: And this concludes this case.
In particular, we have: for any Γ ⊢  : Nat in PCF.This show in particular that we have: Proof.This is a consequence of the two previous lemmas.□ From this simulation, we obtain: Theorem 4. For any term ⊢  : Nat in PCF, there is a term of finitary HEPCF   ≜ let  = ⟦⟧ in ( ()) such that ⊢    : Unit and  (  ) represents the normal form of  (if it exists, otherwise the  (  ) ≡ ⊥).
Proof.Suppose that ⊢  : Nat has no normal form, then ⟦⟧ has no normal form, and thus the effect tree of   is ⊥.Otherwise, suppose that  → * return( ) with  a closed value of type Nat.Then, ⟦⟧ → *  with  ≥ ⟦return( )⟧.In particular,  → * return( ′ ) with  ′ ≥ ⟦ ⟧.Thus, the effect tree of   is the same as ⟦ ⟧.Because  is a closed value of type Nat, this effect tree is the tree representation of the integer  , this concludes the proof.□ Corollary 2. Effect trees produced by HEPCF terms are not MSO-decidable.Indeed, given a PCF term of type Nat, we can translate it into HEPCF and ask if the effect tree of this HEPCF term contains a ⊥, and this would decide if the initial term halts.Thus, this shows that the power of handlers are simply too expressive for MSO model-checking.
Remark.Our proof heavily relies on the fact that handlers can change the output type, which allows here to do this encoding of pairs of computation internally and returns at the end only the second computation.A careful reader might wonder if the control of this output type is necessary for undecidability, and that we could have a decidable MSO model-checking for restrained handlers, with a fixed output type, for example Unit  ⇒  ′ Unit.We show in the next section that this restriction still leads to a language for which MSO model-checking is undecidable.

HANDLERS IN ECPS
In this section, we present a language written directly in CPS-style, with handlers, called HECPS.This language is based on the language described in [Matache 2018;Matache and Staton 2019].The handlers in ECPS seems less expressive than the handlers available in HEPCF especially because we cannot chose the output and input type, however we will show that we still have an undecidability result for HECPS.
Formally, we define HECPS as the language with the following types and grammar: The intuition behind this language is that all terms ,  must be of type , where  is intuitively a return type corresponding to the type of tree in the -Y calculus.In particular, a value of HECPS is either of base type, or a function that outputs in type , described by the type ¬  ( 1 , . . .,  ) Then, the obvious difference with HEPCF is the absence of let • in expressions, because it is captured by a use of continuation, and there is no way to return a value, because we can only stop the computation with a return of type .
The typing rules that differs from HEPCF are given in Figure 9.As for the semantics, it is similar to the one described in Figure 8.The important difference with what we saw before is that we cannot control the input or output type, we need to work with type .Because of this, the previous translation from PCF, that relied strongly on the output type, does not work directly.However, we will again prove that we can encode PCF in this language, and as before we rely on the idea to encode shallow handlers as deep handlers.Formally, a shallow handler is syntactically similar to a deep handler, but the typing rule and semantics are different, as described in Figure 10.We differentiate shallow handlers from usual handlers by adding a † in the handle operation.In the typing derivation the difference is subtle, only the set of effect of the continuation  changes.However, in the semantics the difference is flagrant, the continuation is not called with the handler again.

ECPS with Shallow Handlers
We introduce shallow handlers in order to decompose the translation from PCF, we will first show that we can encode PCF in ECPS with shallow handlers, and then we will see how we can encode shallow handlers using deep handlers in ECPS (the important point being that the proof given in the previous section and in [Hillerström and Lindley 2018] does not work anymore).
Definition 16 (Translation from PCF to ECPS with Shallow Handlers).As in the previous section, we need to encode the infinite type of natural numbers into finite types.We introduce the effect  = { : Unit ⇝ 1}.We define the zero term, denoted Z  by Z  = return.
The translation ⟦•⟧ from PCF to ECPS with shallow handlers is given by And Proof.The proof is straighforard.□ Now, we want to prove that the translation is a correct simulation.
Proof.The only difference with a standard CPS translation is for the case constructor.Thus, only detail this case.
• Consider the reduction case(0, 0 ↦ → ; succ() ↦ → ) → .Then, we have: We need to use the  equivalence to get exactly the result, but we could always formally avoid it, but for the sake of simplicity we just consider that we work with -equivalence.□ From this, we can conclude the undecidability result for ECPS with shallow handlers.
Lemma 10.For any term ⊢  : Nat in PCF, there is a term of ECPS with shallow handlers   ≜ ⟦⟧ (.0) such that ⊢    such that the effect tree of   represents the normal form of  (if it exists, otherwise the effect tree is ⊥).
Proof.Suppose that ⊢  : Nat has no normal form, then ⟦⟧ has no normal form, and thus the effect tree of   is ⊥.Otherwise,  → * return( ) with  a value of type Nat.Then, with the previous lemma, we obtain   → * ⟦return( )⟧ (.0) → * ⟦ ⟧ 0 and we can easily see that the effect tree of ⟦ ⟧ 0 represents exaclty the integer  .□ Thus, to conclude on the undecidability of HECPS, we need to show that we can simulate shallow handlers using deep handlers.

Encoding Shallow Handlers with Deep Handlers
As we are only interested in encoding the previous translation, we fix the set of effect  = { : Unit ⇝ 1}.We introduce a duplication of this effect, and consider the set   = {  : Unit ⇝ 1}.We will use this operation   as a 'marked' effect, similarly to the use of 'marked' letters that we can find in some Turing Machine problems.In the following proof, we will mainly work directly on effect trees, because we will introduce handlers and each of these handlers will slightly transform the computation by just changing a bit the effect tree.We believe this proof is far more intuitive.However, if we want to prove formally a simulation, we would need to use administrative context like before.
As a first step toward this intuition of marked effects, we introduce the marking handlers, denoted   , and described by: Now remark that applying   twice does not change anything, by definition of   .And as we can see with the previous description, the effect tree of with   •   handle  is the same as with   handle  alone.Thus, we can show that the effect tree of with   handle  is the effect tree of  when we replace all  by   except on the root.Thus, we have just found a way to distinguish the root of the effect tree from the rest, and we can use that to simulate a shallow handler.Suppose that we want to simulate the following shallow handler ℎ with a type corresponding to the shallow handlers we used in the previous translation: Consider the following (deep) handler: that has the type ( ∪   ) ⇒  and consider the following term: with  ℎ •   handle .We have: And now we have to show that with  ℎ •   •   handle  behaves like  (the context is administrative).We already know that   •   behave like   , and on effect tree with effect only in   , the handler  ℎ will just replace every   by .Thus, the effect tree of  is exactly the effect tree of with  ℎ •   •   handle .Thus,  ℎ •   behaves like a shallow handler.We can then take the previous encoding of  in ECPS with shallow handlers, and replace the use of a shallow handler ℎ by the use of  ℎ •   , and we obtain a valid encoding of  to ECPS with deep handlers.
Theorem 5. HECPS is not MSO-decidable 10 EPCF WITH RESTRICTED HANDLERS (GENERIC EFFECT) Does the results in the last section mean that we can have algebraic effects but that we have to get away with handlers if HOMC is our concern?Essentially, it rather shows that general handlers, define a transformation on effect trees that is simply too expressive for HOMC.However, this does not mean that simpler tree transformations should not be expressible in HORS, and this section is devoted to introducing a class of handlers whose underlying tree transformations is amenable to HOMC.Intuitively, an interpretation of an algebraic operation  : B ⇝  could be just a function of type  → , with no access to the continuation.This is of course much less expressive than standard handlers, but we will argue that what we obtain is not too restrictive.Formally, we could capture this by a restriction of handlers, defined in the following language, called GEPCF (for Generic Effects PCF): Note that as expected, the handler for an algebraic operation does not have access to the continuation.The static and dynamic semantics is in Figure 11.
Please observe that we e can always simulate a GEPCF handler by a HEPCF handler: Thus, those restrained handlers are a specific case of standard handlers in which the handling of an operation   always has the shape let  =   in ( ).Informally, if we look at it with the point of view of tree transformations, this is exactly what we would obtain from a generic effect for the effect tree monad, that is why we call this language GEPCF.
Let us now show that this language can be translated into the  -calculus, thus proving the MSO-decidability.
Definition 17 (Translation from GEPCF to -Y calculus).We define the translation • ★ on types and terms.For the sake of conciseness, in types, we use some abuse of notations, such as pairs in the left-hand side of an arrow type to denote the obvious curried version of this type.
→  Informally, we will do a CPS-translation in which there is also a handler continuation, of type  ★ , giving an interpretation for all algebraic effects in .We take a  -calculus with a signature Σ with at least all the constants values of EPCF., where =  is the equivalence relation with ((ℎ, ).) ℎ  ≡ .
Proof.The proof goes by induction on the derivation of  → .Some cases are equivalent to the usual CPS translation so we can safely ignore them.Similarly to Lemma 2, we have to take care of some administrative reductions.The new cases correspond to the cases of handlers.The contextual case is straightforward since we can reduce the head of the  −  -term.The most interesting case is the handlers against an algebraic operation.For this case, we have We can now define a canonical continuation handler for a given set of effects.
With this remark, we can conclude with the following theorem: Theorem 6.For any ⊢   :  , for any continuation ⊢  :  ★ → , we have: Proof.We can proceed by induction on the effect tree of .
• If  can be reduced infinitely, then by simulation,  ★ ℎ   can be reduced infinitely too.
• If  → *  ( ;  .).Then  ★ ℎ   → *    ★ ( ★ [ And we can conclude by induction hypothesis.□ In particular, if we have  has a base type and a continuation that is the encoding of the identity, we can recover exactly the initial effect tree.Thus, we have: Corollary 3.For any GEPCF computation ⊢  :  and ⊢  :  ★ → , the MSO model-checking on  () [return( ) ←  (  * )] is decidable.
Similarly to what happened in Corollary 1, there are of course some degrees of freedom in the choice of the continuation .

RELATED WORK
Effectful Higher-Order Programs and their Verification.This is definitely not the first paper concerned with the verification of higher-order effectful programs.the denotational semantics of calculi having this nature has been studied since Moggi's seminal work on monads [Moggi 1988], thus implicitly providing notions of equivalence and refinement.All this has been given a more operational flavor in Plotkin and Power's account on adequacy for algebraic effects [Plotkin and Power 2003], from which the operational semantics presented in this paper is greatly inspired.Logics for algebraic effects have been introduced by Pretnar and Plotkin [Plotkin and Pretnar 2008], by Matache and Staton [Matache and Staton 2019], and by Simpson and Voorneveld [Simpson and Voorneveld 2019].The latter has certainly been another major source of inspiration although, as explicitly stated by the authors 1 , automatic verification techniques are to be considered out of the scope.In fact, we are not aware of any attempt to study the decidability of the aforementioned theories.
1 In [Simpson and Voorneveld 2019], Section 10, paragraph 8: "We view the infinitary propositional logic of this paper as providing a low-level language, into which practical high-level finitary logics for expressing program properties can potentially be compiled.[...] We view the development of such high-level logics and their compositional reasoning principles, aimed at practical specification and verification, as a particularly promising topic for future research." Higher-Order Model Checking.Model checking of higher-order programs has been an active topic of investigation in the last twenty years, with many positive results, staring from the pioneering and partial results by Knapik et al.[Knapik et al. 2001, 2002] to the Ong's already mentioned breakthrough result [Ong 2006], followed by Kobayashi and co-authors' work on model checking as (intersection) type checking [Kobayashi and Ong 2009].Noticeably, some of these works go in the direction of extending the aforementioned decidability results to higher-order calculi endowed with some specific form of effect, like probabilistic choice [Kobayashi et al. 2020] or some form of resource usage [Kobayashi 2009].Outcomes are not always been on the positive side, as the undecidability of the model checking problem for probabilistic variations on HORSs shows.Again, we are not aware of any study aimed at giving general criteria for decidability.
Effect Handlers.Effect handlers are a linguistic feature allowing to give computational meaning to algebraic operations through handlers, i.e. routines specifically dedicated to the management of these effects, which in this way can be managed internally by the program itself rather than by the environment.Given their elegance and naturalness in generalizing standard language construction like the try with operator for exceptions, handlers has been largely studied both theoretically and concretely [Bauer 2019;Biernacki et al. 2019;Hillerström and Lindley 2018;Hillerström et al. 2017;Kammar et al. 2013;Plotkin and Pretnar 2009;Sekiyama et al. 2020].We are not aware of any study about handlers in a finitary setting, and even less about questions of decidability with regards to higher-order model checking.

CONCLUSION
This paper tackles, for the first time in a general way, the problem of evaluating the intrinsic difficulty of the higher-order model checking problem when applied to programs that exhibit effects, possibly managed through handlers.The results obtained are in two styles: while the problem of capturing algebraic operations in calculi amenable to HOMC does not pose particular problems and indeed can be solved in its generality, observing the effects produced by such operations and handling them must be done with great care: we observe that in general this leads to undecidability, but that in both cases, criteria can be defined allowing to keep the problem decidable.This consists, respectively, in observing the effects so that this observation can be expressed in MSO and in a restricted class of handlers sufficient for the modeling of the so-called generic effects.
There are aspects that this paper deliberately overlooks, but which certainly deserve further study.First of all, it should be mentioned that the impact of effects on (known) complexity results about HOMC is not studied in detail here, but that the introduced translations (in particular those in Section 5 and Section 8) could be implemented more efficiently, following works on order optimization [Nakamura et al. 2020].It should also be said that the possibility of introducing logics more powerful than MSO this way capturing quantitative observations without possibly falling back into the known cases of undecidability (e.g., in the case of probabilistic effects) is a very interesting direction that the authors intend to investigate in the immediate future.