Trace data loss in worker processes

Weave uploads trace data in background threads to minimize impact on your application’s performance. However, you might lose trace data when you use a multiprocessing or task queue system, or a worker process like Celery. This happens if the worker process exits before background threads finish uploading traces.

To prevent data loss in worker processes, ensure that background uploads complete by calling client.flush() or client.finish() before the worker task completes. This helps in short-lived environments like AWS Lambda, Google Cloud Run, or notebooks that exit right after execution.

These methods have different purposes, so choose the one that fits your environment:

• weave.flush(): Provides silent flushing. Recommended when Weave is integrated into worker processes or continuous integration (CI) environments.
• weave.finish(): Provides progress feedback with a progress bar or status callbacks. Recommended for interactive scripts or notebooks.

Both methods block until all background uploads complete, which ensures that no trace data is lost when the worker process exits.

The following example shows how to call client.finish() from a Celery task to flush traces before the worker returns. Replace [TEAM-NAME] with your W&B team name and [PROJECT-NAME] with your W&B project name:

from celery import Celery
import weave

app = Celery('tasks')

@app.task
def process_task(input_data):
    weave.init("[TEAM-NAME]/[PROJECT-NAME]")

    try:
        # Your task logic that creates traces
        result = your_processing_function(input_data)

        # Ensure that all traces are uploaded before the task completes
        weave.finish()

        return result
    finally:
        pass

As an alternative, you can use the with context manager to call weave.finish() automatically on exit:

with weave.init("[TEAM-NAME]/[PROJECT-NAME]") as client:
    result = your_processing_function(input_data)

    return result

You can also improve the performance of your application with weave.flush(). For more information, see Flushing.

For additional resilience against data loss, consider enabling the write-ahead log, which writes trace data to disk before it sends the data to the server.

---

Trace Data Performance