ERROR: failed to solve: circular dependency detected on stage: weights

[gurteshwar]

Hello,

Trying to push a model with --separate-weights fails with this error. Works fine when the flag is not used. I am using cog version 0.9.7. I also tried deleting .cog , removed the models/ folder from the auto-generated .dockerignore

Building Docker image from environment in cog.yaml as r8.im/xxxxx/xxxxx...
[+] Building 187.8s (8/8) FINISHED                                                                                                                                                                                                                                            docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                                                                                                                                                             0.0s
 => => transferring context: 1.47kB                                                                                                                                                                                                                                                           0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                          0.0s
 => => transferring dockerfile: 141B                                                                                                                                                                                                                                                          0.0s
 => resolve image config for docker.io/docker/dockerfile:1.4                                                                                                                                                                                                                                  1.1s
 => CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                                                             0.0s
 => [internal] load build context                                                                                                                                                                                                                                                            98.2s
 => => transferring context: 10.62GB                                                                                                                                                                                                                                                         98.2s
 => [1/1] COPY models /src/models                                                                                                                                                                                                                                                            58.0s
 => preparing layers for inline cache                                                                                                                                                                                                                                                        30.1s
 => exporting to image                                                                                                                                                                                                                                                                        0.0s
 => => exporting layers                                                                                                                                                                                                                                                                       0.0s
 => => writing image sha256:5be97ec87e2fb8b2f8f001848c916dd1a2031b3624ff2974d14e8558adc3c655                                                                                                                                                                                                  0.0s
 => => naming to r8.im/xxxxxx/xxxxxx                                                                                                                                                                                                                                    0.0s
[+] Building 3.7s (5/5) FINISHED                                                                                                                                                                                                                                              docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                          0.0s
 => => transferring dockerfile: 2.47kB                                                                                                                                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                             0.1s
 => => transferring context: 1.49kB                                                                                                                                                                                                                                                           0.0s
 => resolve image config for docker.io/docker/dockerfile:1.4                                                                                                                                                                                                                                  3.4s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                                                                                                                              0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                                                             0.0s
Dockerfile:1
--------------------
   1 | >>> #syntax=docker/dockerfile:1.4
   2 |     FROM python:3.11 as deps
   3 |     COPY .cog/tmp/build3627906443/cog-0.0.1.dev-py3-none-any.whl /tmp/cog-0.0.1.dev-py3-none-any.whl
--------------------
ERROR: failed to solve: circular dependency detected on stage: weights
ⅹ Failed to build runner Docker image: Failed to build Docker image: exit status 1

Below is cog.yaml

# Configuration for Cog ⚙️
# Reference: https://cog.run/yaml

build:
  # set to true if your model requires a GPU
  gpu: true

  # a list of ubuntu apt packages to install
  system_packages:
    - "git"

  # python version in the form '3.11' or '3.11.4'
  python_version: "3.11"
  python_requirements:  requirements.txt

# predict.py defines how predictions are run on your model
predict: "predict.py:Predictor"

[hervenivon]

Same issue here with 0.9.7 and 0.9.8 on Linux.

[Adrian-Serna]

I have the same problem on mac with cog version 0.9.8. Also try with the versions v0.9.9-alpha and v0.10.0-alpha7 and I only got new problems. Yesterday I was talking to Replicate technical support and they usually communicated the problem to the cog developers.

[gurteshwar]

Hmm, seems like this isn't well maintained. No way to run custom models with everything baked into container without this :/ I guess this blocks any further evaluation of replicate as a provider

[Adrian-Serna]

This has been the response I have had.

" This is a known huge friction point for pushing/building Cog models currently. I've flagged the issues with --separate-weights to the devs on the team that maintain Cog to prioritize a fix for it, but I won't have an ETA to share on when they'll be able to land that work.

I'm sorry about that. I know it's an impediment to the workflows for iterating on models. I promise we're thinking about how to solve it!

Thanks for your patience while we work out the kinks in the product."

What makes Replicate unfeasible for developing a model with heavy dependencies without a local GPU. The option remains to work with a rented server for development and upload the implementation to Replicate only when it is ready.