Envoy Gateway: Help needed - Coverage build is failing after using `compile_native_go_fuzzer_v2`

[sudiptob2]

After using the compile_native_go_fuzzer_v2, we encountered some issues with the fuzzing build. These were resolved using the suggestion provided here: google/oss-fuzz#14190. Thanks for the help to resolve that! However, the coverage build is still failing.

Reference: Build logs

Relevant log excerpt:

Step #4: gcr.io/oss-fuzz-base/base-runner:latest
Finished Step #4
Starting Step #5
Step #5: Already have image (with digest): gcr.io/oss-fuzz-base/base-runner
Step #5: [/corpus/FuzzGatewayAPIToXDS.zip]
Step #5:   End-of-central-directory signature not found.  Either this file is not
Step #5:   a zipfile, or it constitutes one disk of a multi-part archive.  In the
Step #5:   latter case the central directory and zipfile comment will be found on
Step #5:   the last disk(s) of this archive.
Step #5: unzip:  cannot find zipfile directory in one of /corpus/FuzzGatewayAPIToXDS.zip or
Step #5:         /corpus/FuzzGatewayAPIToXDS.zip.zip, and cannot find /corpus/FuzzGatewayAPIToXDS.zip.ZIP, period.
Step #5: Failed to unpack the corpus for FuzzGatewayAPIToXDS. This usually means that corpus backup for a particular fuzz target does not exist. If a fuzz target was added in the last 24 hours, please wait one more day. Otherwise, something is wrong with the fuzz target or the infrastructure, and corpus pruning task does not finish successfully.

It appears that the coverage build is looking for a ZIP file for the corpus that does not exist.
We are currently using the FuzzGatewayAPIToXDS_seed_corpus.zip naming pattern for the seed corpus file, which previously worked without issues.

Looking for guidance on the possible cause of this problem.

[arkodg]

ptal @AdamKorcz @DavidKorczynski

[AdamKorcz]

This typically is a sign that the fuzzer is not running and not generating corpus. One reason for this is that the fuzzer is broken - it doesn't seem to be based on running it locally - or that OSS-Fuzz just hasn't triggered a run yet. The fuzzer may not have run after you modified the build script. If you check "Fuzzer stats/coverage" in oss-fuzz.com and select a timeframe that starts after you changed the build script, can you see whether the fuzzer has run (the "tests_executed" is a good indicator).

[sudiptob2]

@AdamKorcz Thanks for the guidance so far. I did some deeper investigation into the coverage-build failure, and here’s a summary of what I found:

The root cause seems to be the empty corpus backup. In GCS, public.zip for gateway_FuzzGatewayAPIToXDS is empty, ref: Bucket link. As a result, the coverage build attempts to unpack a non-existent backup corpus and fails.

The fuzzer is running successfully on OSS-Fuzz and generating inputs. The live corpus is ~700MB. The issue seems to be that the corpus pruning task did not complete, so no pruned corpus was published to the GCS Bucket.

I downloaded the corpus locally and ran the pruning task, and I am able to generate a minimized corpus following the steps below:

1. Download the corpus

2. python3 infra/helper.py build_fuzzers --sanitizer=address gateway

3. Pruning: ./fuzzer_binary -merge=1 -timeout=<duration> <pruned_corpus_dir> <downloaded_corpus_dir>
		OUPUT:
		#62789  DONE   cov: 37125 ft: 109933 exec/s: 13 rss: 960Mb
		MERGE-OUTER: successful in 1 attempt(s)
		MERGE-OUTER: the control file has 5801745 bytes
		MERGE-OUTER: consumed 5Mb (208Mb rss) to parse the control file
		MERGE-OUTER: 4990 new files with 109933 new features added; 37125 new coverage edges
		./build/out/gateway/FuzzGatewayAPIToXDS -merge=1 -timeout=100 minimized_corpu  4824.49s user 15.68s system 101% cpu 1:19:50.82 total

After that, I ran python3 infra/helper.py coverage --fuzz-target=<fuzz_target_name> $PROJECT_NAME with the minimized corpus, which generates coverage correctly.

So, the fuzzer itself appears to be working as expected. The problem is that the OSS-Fuzz pruning step is not completing (maybe due to a timeout; it takes about 1hr20min on my local machine), resulting in an empty backup corpus and causing the coverage build to fail.

Is there a way to view pruning-task logs from the https://oss-fuzz.com/ UI? Happy to try any additional debugging steps on my end.