I think this isn't picked up by the usual "reference is not allowed" because the recursive reference to what is under a lambda in this situation.

Does anyone have any thoughts as to how we could prevent this sort of construction? Just based on the generated code, it's not clear that there's a foolproof way to prevent it. But maybe something can be done when specialized to instance construction?

OCaml is another language with recursive bindings and strict evaluation. It has special rules to prevent exactly this problem. They are documented in detail in http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#s%3aletrecvalues.

I think that the gist of it is that the RHS cannot be a function application expression if the binding is a free variable of that expression.

Here's a simpler reproducible example:

undefined = (\_ -> undefined) unit

I think it would be nice to solve this because it can cause nasty bugs that aren't caught statically. However it would be a breaking change.

I don't think those rules cover this case since there are no direct applications in the recursive binding group.

I have another (pretty contrived) example of this:

module Main where

import Prelude
import Effect (Effect)
import Effect.Console (log)

class CA v a where
  lengthA :: forall m . v m a -> Int

class (CA w a) <= CB v w a | v a -> w, w a -> v where
  lengthB :: v a -> Int
  fromA :: forall m . w m a -> v a
  toA   :: forall m . v a -> w m a

defaultLengthB :: forall v w a . (CB v w a) => v a -> Int
defaultLengthB v = lengthA $ toA v

data DA m a = DA Int
data DB a = DB Int

instance caDA :: CA DA a where
  lengthA (DA i) = i

instance cbDB :: (CA DA a) => CB DB DA a where
  lengthB = defaultLengthB
  fromA (DA a) = (DB a)
  toA (DB a) = (DA a)

main :: Effect Unit
main = log (show (lengthB (fromA (DA 3))))

produces the output:

* Building project in /tmp/rec
* Build successful.
/tmp/rec/output/Main/index.js:52
var cbDB = function (dictCA) {
                    ^

RangeError: Maximum call stack size exceeded
    at cbDB (/tmp/rec/output/Main/index.js:52:21)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
    at cbDB (/tmp/rec/output/Main/index.js:57:23)
* ERROR: Subcommand terminated with exit code 1

Like in #2975, changing lengthB = defaultLengthB to lengthB x = defaultLengthB x fixes the issue.

Yet another example:

module Main where

import Prelude
import Effect (Effect)
import Data.Array (head)
import Data.Maybe (Maybe)

class Klass a where
  foo :: a

instance klassUnit :: Klass (Unit) where
  foo = unit

data M k v
  = Tip
  | Bin k v

instance klassM :: (Klass k, Klass v) => Klass (M k v) where
  foo = Tip

data Trie c m
  = Trie (M c (Trie c m))

instance klassTrie :: (Klass c, Klass m) => Klass (Trie c m) where
  foo = Trie foo

hd :: forall a. Klass a => Array a -> Maybe a
hd = head

main :: Effect (Maybe (Trie Unit Unit))
main = pure $ hd []

It blows up the stack. The culprit:

var klassTrie = function (dictKlass) {
    return function (dictKlass1) {
        return new Klass(new Trie(foo(klassM(dictKlass)(klassTrie(dictKlass)(dictKlass1)))));
    };
};

But if the definition of klassTrie is changed to something equivalent:

instance klassTrie :: (Klass c, Klass m) => Klass (Trie c m) where
  foo = Trie Tip

everything is fine:

var klassTrie = function (dictKlass) {
    return function (dictKlass1) {
        return new Klass(new Trie(Tip.value));
    };
};

Besides the typeclass self-reference issue, I suspect there's something else at play here.

Shouldn't the generated code be the same in both cases? After all, if plugging in type hole foo = Trie ?foo, the compiler correctly infers the type to be M c0 (Trie c0 m1). But the generated code seems to be sub-optimal thus triggers the self-referencing.