Progress log of image generation not working

[921kiyo]

There was a discussion in #152 that Prediction class instance now provides progress information. But I don't thinks it's working as expected. Could you please have a look at this and confirm that progress property is always available for Prediction instance?

Also there used to progress info in the log property of Prediction class instance, but I don't get this anymore (I was relying on this to check the progress in our UI)

[mattt]

Hey @921kiyo. Thanks for opening up a new issue for this.

The way progress is determined is by scanning the prediction logs and scanning for output in a given pattern. If the logs aren't present or not in the right format, the progress property will return None.

In a recent issue over on the JS client (replicate/replicate-javascript#146 (comment)), someone was reporting a similar issue with their mode, and it turned out to be an issue with logs not populating for their predictions. That was also a fine-tune of SDXL. That issue appeared to be resolved, but perhaps it's come up again. I'll ask around to see what's going on.

[simonpicard]

Hi @mattt, I am working on the same codebase as Kiyo and I can confirm that logs are not available when using replicate.predictions.get on a specific job.

Those jobs are created via replicate.predictions.create. For reference, here are the typical input we use:

{
  "width": 1024,
  "height": 1024,
  "prompt": "building",
  "num_outputs": 2
}

While the job is running, the logs are not available on the web UI either, see attached screenshot.

Once the job is completed, the logs are available both using the API and the web UI. Here is the results:

Using seed: 42492
Ensuring enough disk space...
Free disk space: 1462543171584
Downloading weights: https://pbxt.replicate.delivery/[SOME_UUID]/trained_model.tar
b'Downloaded 186 MB bytes in 0.509s (366 MB/s)\nExtracted 186 MB in 0.071s (2.6 GB/s)\n'
Downloaded weights in 0.9005029201507568 seconds
Loading fine-tuned model
Does not have Unet. assume we are using LoRA
Loading Unet LoRA
Prompt: building
txt2img mode
  0%|          | 0/50 [00:00<?, ?it/s]/usr/local/lib/python3.9/site-packages/torch/nn/modules/conv.py:459: UserWarning: Applied workaround for CuDNN issue, install nvrtc.so (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:80.)
return F.conv2d(input, weight, bias, self.stride,
  2%|▏         | 1/50 [00:01<00:57,  1.18s/it]
  4%|▍         | 2/50 [00:01<00:38,  1.25it/s]
  6%|▌         | 3/50 [00:02<00:31,  1.47it/s]
  8%|▊         | 4/50 [00:02<00:28,  1.61it/s]
 10%|█         | 5/50 [00:03<00:26,  1.69it/s]
 12%|█▏        | 6/50 [00:03<00:25,  1.74it/s]
 14%|█▍        | 7/50 [00:04<00:24,  1.78it/s]
 16%|█▌        | 8/50 [00:04<00:23,  1.81it/s]
 18%|█▊        | 9/50 [00:05<00:22,  1.83it/s]
 20%|██        | 10/50 [00:06<00:21,  1.84it/s]
 22%|██▏       | 11/50 [00:06<00:21,  1.85it/s]
 24%|██▍       | 12/50 [00:07<00:20,  1.86it/s]
 26%|██▌       | 13/50 [00:07<00:19,  1.86it/s]
 28%|██▊       | 14/50 [00:08<00:19,  1.86it/s]
 30%|███       | 15/50 [00:08<00:18,  1.87it/s]
 32%|███▏      | 16/50 [00:09<00:18,  1.87it/s]
 34%|███▍      | 17/50 [00:09<00:17,  1.87it/s]
 36%|███▌      | 18/50 [00:10<00:17,  1.87it/s]
 38%|███▊      | 19/50 [00:10<00:16,  1.87it/s]
 40%|████      | 20/50 [00:11<00:16,  1.87it/s]
 42%|████▏     | 21/50 [00:11<00:15,  1.87it/s]
 44%|████▍     | 22/50 [00:12<00:14,  1.87it/s]
 46%|████▌     | 23/50 [00:12<00:14,  1.87it/s]
 48%|████▊     | 24/50 [00:13<00:13,  1.86it/s]
 50%|█████     | 25/50 [00:14<00:13,  1.86it/s]
 52%|█████▏    | 26/50 [00:14<00:12,  1.87it/s]
 54%|█████▍    | 27/50 [00:15<00:12,  1.87it/s]
 56%|█████▌    | 28/50 [00:15<00:11,  1.87it/s]
 58%|█████▊    | 29/50 [00:16<00:11,  1.87it/s]
 60%|██████    | 30/50 [00:16<00:10,  1.86it/s]
 62%|██████▏   | 31/50 [00:17<00:10,  1.87it/s]
 64%|██████▍   | 32/50 [00:17<00:09,  1.86it/s]
 66%|██████▌   | 33/50 [00:18<00:09,  1.86it/s]
 68%|██████▊   | 34/50 [00:18<00:08,  1.86it/s]
 70%|███████   | 35/50 [00:19<00:08,  1.85it/s]
 72%|███████▏  | 36/50 [00:19<00:07,  1.85it/s]
 74%|███████▍  | 37/50 [00:20<00:07,  1.85it/s]
 76%|███████▌  | 38/50 [00:21<00:06,  1.85it/s]
 78%|███████▊  | 39/50 [00:21<00:05,  1.86it/s]
 80%|████████  | 40/50 [00:22<00:05,  1.86it/s]
 82%|████████▏ | 41/50 [00:22<00:04,  1.86it/s]
 84%|████████▍ | 42/50 [00:23<00:04,  1.86it/s]
 86%|████████▌ | 43/50 [00:23<00:03,  1.86it/s]
 88%|████████▊ | 44/50 [00:24<00:03,  1.86it/s]
 90%|█████████ | 45/50 [00:24<00:02,  1.85it/s]
 92%|█████████▏| 46/50 [00:25<00:02,  1.85it/s]
 94%|█████████▍| 47/50 [00:25<00:01,  1.85it/s]
 96%|█████████▌| 48/50 [00:26<00:01,  1.85it/s]
 98%|█████████▊| 49/50 [00:26<00:00,  1.85it/s]
100%|██████████| 50/50 [00:27<00:00,  1.85it/s]
100%|██████████| 50/50 [00:27<00:00,  1.82it/s]

As explained by Kiyo, we use the logs to fetch the progress % and display it on our UI to our users.

Many thanks for your investigation, please keep us posted.