Predictions often fail on meta/llama-2-70b

[jdkanu]

Calls to meta/llama-2-70b are sometimes succeeding, but sometimes failing. It is very unreliable.

This is the code

output = replicate.run(
        "meta/llama-2-70b",
        input={
            "prompt": "Q: Would a pear sink in water? A: Let's think step by step. ",
            "max_new_tokens": 10000,
            "temperature": 0.01,
        }
    )

Example failure: https://replicate.com/p/x3brrjtbwq4ky6zm2z2ay27amy
Example failure: https://replicate.com/p/72pdpvtby7l7wgdzrpzzldqzne
Example failure: https://replicate.com/p/ucbimbtbhyjrw5udypzf6srsm4
Example success: https://replicate.com/p/n6hg2cdbym5ksifmlm6yfahjzm
Example success: https://replicate.com/p/j2jkwn3bfl6w2wc4q53mwju2o4
Example success: https://replicate.com/p/mtb6jcrbzjh57s2wjzfycegxoa

[mattt]

Hi @jdkanu. Thank you for reporting this. Looking at our telemetry, it does seem like predictions on GPUs in certain regions are failing more often due to read timeouts. We're investigating the cause, and working on a remediation.

[jdkanu]

Thank you