Troubleshooting GitHub Actions Cache: 404 Upload Not Found

by Axel Sørensen 59 views

Hey everyone! Today, I want to dive deep into a specific error I encountered while stress-testing the GitHub Actions cache server: "FinalizeCacheEntryUpload 404 Upload Not Found." This can be a real head-scratcher, especially when things seem to work fine intermittently. Let’s break down what this error means, what might be causing it, and how to troubleshoot it.

Decoding the Error Message

First off, let’s take a closer look at the error messages we’re dealing with. The error appears both in the GitHub Actions logs and on the server side, giving us clues from both ends. This is how the error manifests in GitHub Actions:

Attempt 1 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 3000 ms...
Attempt 2 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 5374 ms...
Attempt 3 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 8804 ms...
Warning: Failed to save: Failed to FinalizeCacheEntryUpload: Received non-retryable error: Failed request: (404) Upload not found

And here’s how it looks on the server:

[cache-server-node-1] ⚙ Request: POST /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload

[cache-server-node-1]  ERROR  Response: POST /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload > 404
 Upload not found

   at createError$1 (server/index.mjs:647:15)
   at Object.handler (server/chunks/routes/twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload.mjs:46:11)
   at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
   at async Object.handler (server/index.mjs:1633:19)
   at async Server.toNodeHandle (server/index.mjs:1904:7)

 [cause]: { statusCode: 404, statusMessage: 'Upload not found' }

The critical part here is the "404 Upload not found" error. A 404 error generally means that the server couldn't find the resource you were looking for. In this context, it suggests that the cache server can’t locate the upload it's supposed to finalize. This error usually occurs during the FinalizeCacheEntryUpload process, which is the last step in saving a cache entry after the upload is complete.

Why Does This Happen?

There are several reasons why you might encounter this error. Understanding these potential causes is crucial for effective troubleshooting:

  1. Timing Issues and Race Conditions: One of the most common reasons for a 404 error in this scenario is a timing issue. Think of it like this: the server receives a request to finalize the upload before the upload process is fully complete or before the server has properly registered the uploaded data. This can happen in distributed systems where there might be a slight delay in synchronizing data across different nodes or services. In high-load situations, these timing issues can become more frequent.

  2. Data Inconsistencies: Sometimes, the metadata about the upload might not be consistent between different parts of the caching system. For example, the information about the uploaded file's location or status might not be correctly updated in the database or storage backend. This inconsistency can lead to the "Upload not found" error because, from the server's perspective, the upload never happened or doesn't exist at the expected location.

  3. Network Issues and Intermittent Errors: Network glitches or temporary connectivity problems can also cause this error. If the connection between the GitHub Actions runner and the cache server is interrupted during the upload process, the server might not receive all the necessary data or might not be able to confirm the upload's completion. This is more likely to occur in environments with unstable network connections.

  4. Large Cache Sizes and Upload Timeouts: When dealing with large caches, the upload process can take a significant amount of time. If the server or the client has aggressive timeout settings, the upload might be prematurely terminated, leading to a 404 error when the finalize request is sent. This is particularly relevant in the reported case, where the user was stress-testing with a 10GB cache.

  5. Server-Side Bugs or Misconfigurations: Lastly, there could be bugs in the cache server software itself or misconfigurations in the server setup. These issues might not be immediately obvious and could require a deeper investigation of the server logs and configuration settings. For example, incorrect storage paths, database connection issues, or errors in the server’s request handling logic could all contribute to this problem.

Reproducing the Issue

To effectively troubleshoot this error, it's important to understand how to reproduce it. In the initial report, the user was able to reproduce the issue using a GitHub Actions workflow that generates a large cache. Here’s the relevant part of the workflow:

ci:
 name: Run CI
 runs-on: arc-stg-amd64
 steps:
 # At this step it will attempt to restore foo from the cache server (if the corresponding key is found)
 - name: Cache foo directory
 uses: actions/cache@v4
 with:
 path: foo
 key: test-foo-${{ runner.os }}-${{ github.run_id }}
 restore-keys: |
 test-foo-${{ runner.os }}-

 - name: Display contents after restore
 run: |
 echo "Contents of foo:"
 ls -l foo || echo "No cache found"

 # Generate a total of 10GB files under 'foo' to create a large cache for the stress test
 - name: Generate large cache files
 run: |
 mkdir -p foo
 for i in {1..100}; do
 head -c 100M </dev/urandom > foo/artifact_$i.bin
 done

This workflow does the following:

  1. Restores the cache: It attempts to restore a cache for the foo directory using the actions/cache@v4 action.
  2. Displays contents: It checks and displays the contents of the foo directory after the restore attempt.
  3. Generates large cache files: It creates 100 files, each 100MB in size, inside the foo directory, resulting in a 10GB cache. This step is crucial for simulating a large cache upload and triggering the issue.

By creating a large cache, the workflow increases the likelihood of encountering timing issues or timeouts during the upload process, thereby reproducing the "FinalizeCacheEntryUpload 404 Upload Not found" error.

Troubleshooting Steps

Now that we understand the error and its potential causes, let’s discuss how to troubleshoot it. Here’s a step-by-step approach you can take:

1. Check Server Logs

The first thing you should do is examine the server logs. The logs often contain detailed information about what’s happening on the server side, including any errors or warnings that might provide clues. In this case, the server log snippet already gives us the "404 Upload not found" error, but it doesn’t tell us why the upload was not found.

Look for any other error messages or warnings that precede the 404 error. These might indicate issues with storage, database connections, or other server-side problems. Pay attention to timestamps and correlate them with the GitHub Actions logs to understand the sequence of events.

2. Review Network Configuration

Network issues can often lead to 404 errors. Ensure that there are no network connectivity problems between the GitHub Actions runner and the cache server. Check for firewalls, proxies, or other network devices that might be interfering with the connection.

  • DNS Resolution: Verify that the runner can correctly resolve the cache server’s hostname.
  • Firewall Rules: Ensure that the firewall allows traffic between the runner and the cache server on the necessary ports.
  • Proxy Settings: If you’re using a proxy, make sure it’s correctly configured and that it’s not causing any issues with the connection.

3. Examine Cache Server Configuration

Misconfigurations in the cache server setup can also lead to this error. Review the server’s configuration files and settings to ensure everything is correctly set up.

  • Storage Configuration: Check the storage backend configuration (e.g., S3, Azure Blob Storage) to ensure that the server has the correct credentials and access permissions.
  • Database Configuration: Verify the database connection settings (if you’re using a database) to ensure that the server can connect to the database and that the database is running correctly.
  • Timeout Settings: Review the server’s timeout settings to ensure they’re appropriate for large uploads. If the timeouts are too short, the server might prematurely terminate the upload, leading to a 404 error.

4. Investigate Timing Issues and Race Conditions

As mentioned earlier, timing issues and race conditions are a common cause of this error. To investigate this, you might need to add some logging or debugging to your cache server code.

  • Log Upload Start and End Times: Add logs to record when an upload starts and when it completes. This can help you identify if uploads are taking longer than expected or if there are any delays in the upload process.
  • Track Metadata Updates: Log when metadata about the upload is updated in the database or storage backend. This can help you identify inconsistencies in the metadata.
  • Implement Retries: If you suspect timing issues, you might consider implementing retries in your client code. If a 404 error is received, retry the finalize request after a short delay. This can help mitigate transient issues.

5. Check for Large File Handling Issues

Since the error was reproduced with a large cache (10GB), it’s important to check for issues related to large file handling.

  • File Size Limits: Ensure that there are no file size limits imposed by the cache server, storage backend, or network infrastructure.
  • Multipart Uploads: If you’re using a storage backend like S3, make sure that the server is correctly handling multipart uploads. Multipart uploads allow you to split large files into smaller chunks, which can improve performance and reliability.
  • Buffering and Memory Issues: Check for any buffering or memory issues on the server side. Large uploads can consume a significant amount of memory, and if the server is not configured correctly, it might run out of memory, leading to errors.

6. Update Dependencies and Software

Outdated dependencies or software versions can sometimes contain bugs that cause unexpected errors. Make sure that your cache server software, libraries, and dependencies are up to date.

  • Cache Server Software: Check for updates to the cache server software and apply any patches or fixes that might address the issue.
  • Libraries and Dependencies: Update any libraries or dependencies used by the cache server to the latest versions.

7. Monitor Server Performance

Monitor the performance of your cache server to identify any resource bottlenecks or performance issues. Use monitoring tools to track metrics like CPU usage, memory usage, disk I/O, and network traffic.

  • CPU and Memory: High CPU or memory usage can indicate that the server is under heavy load and might be struggling to handle requests.
  • Disk I/O: High disk I/O can indicate that the server is spending a lot of time reading from or writing to disk, which can slow down performance.
  • Network Traffic: High network traffic can indicate that the server is handling a lot of requests, which can also lead to performance issues.

8. Reproduce in a Controlled Environment

If possible, try to reproduce the issue in a controlled environment. This can help you isolate the problem and identify the root cause more easily.

  • Minimal Setup: Try to reproduce the issue with a minimal setup, using only the necessary components. This can help you rule out any interference from other services or configurations.
  • Consistent Steps: Follow a consistent set of steps to reproduce the issue. This can help you ensure that you’re triggering the same error every time.

Addressing the Intermittent Success

One of the confusing aspects of this issue is that the cache sometimes works, and files can be downloaded successfully. This intermittency suggests that the problem is not a complete failure but rather a sporadic issue, likely related to timing or resource contention.

The fact that a new GitHub Action can sometimes find and download the cached files, despite the "FinalizeCacheEntryUpload" error, indicates that the underlying data might still be present, but the finalization process is failing. This further points to potential race conditions or timing issues in the server’s handling of the upload process.

Conclusion

The "FinalizeCacheEntryUpload 404 Upload Not found" error on a GitHub Actions cache server can be a complex issue to troubleshoot. However, by understanding the potential causes—timing issues, data inconsistencies, network problems, large file handling, and server misconfigurations—you can systematically investigate and resolve the problem.

Remember to start by checking the server logs, reviewing your network configuration, and examining your cache server settings. If the issue persists, delve deeper into timing issues, large file handling, and server performance. By following these steps, you’ll be well-equipped to tackle this error and ensure the reliable operation of your GitHub Actions cache server.

I hope this detailed guide helps you guys in troubleshooting this issue. If you have any more questions or insights, feel free to share them in the comments below! Let’s keep the discussion going and help each other out.