[Bug]: CustomAnalyzerTest tests fail on macOS 13 and 14

[jrdodds]

Issue Description

Recently PR #10076 added a new CustomAnalyzerTest [Theory] with two sets of data. The PR build checks passed running on a macos-12 image.

However, when I build locally and run tests on macOS 14 on ARM and macOS 13 on Intel, the tests fail.

Steps to Reproduce

1. Run ./build.sh.
2. Run ./build.sh --test

Expected Behavior

All tests should pass.

Actual Behavior

CustomAnalyzerTest tests fail.

Failed tests
Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(customAnalyzerNames: [\"CustomAnalyzer\", \"CustomAnalyzer2\"], analysisCandidate: \"AnalysisCandidateWithMultipleAnalyzersInjected\", expectedRegisteredRules: [\"CustomRule1\", \"CustomRule2\", \"CustomRule3\"])
Shouldly.ShouldAssertException : success\r\n    should be\r\nTrue\r\n    but was\r\nFalse
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.BuildAnalyzerRules(TestEnvironment env, String[] customAnalyzerNames) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 174
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(String[] customAnalyzerNames, String analysisCandidate, String[] expectedRegisteredRules) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 145
   at InvokeStub_EndToEndTests.CustomAnalyzerTest(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(customAnalyzerNames: [\"CustomAnalyzer\"], analysisCandidate: \"AnalysisCandidate\", expectedRegisteredRules: [\"CustomRule1\", \"CustomRule2\"])
Shouldly.ShouldAssertException : success\r\n    should be\r\nTrue\r\n    but was\r\nFalse
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.BuildAnalyzerRules(TestEnvironment env, String[] customAnalyzerNames) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 174
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(String[] customAnalyzerNames, String analysisCandidate, String[] expectedRegisteredRules) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 145
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

The Microsoft.Build.BuildCheck.UnitTests_net8.0_x64.html and .Microsoft.Build.BuildCheck.UnitTests_net8.0_x64xml log files are attached in a .zip.

Microsoft.Build.BuildCheck.UnitTests_net8.0_x64.zip

Analysis

The tests fail in BuildAnalyzerRules() after calling RunnerUtilities.ExecBootstrapedMSBuild():

msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs

Lines 164 to 174 in 393c2fe

	private IList<string> BuildAnalyzerRules(TestEnvironment env, string[] customAnalyzerNames)
	{
	var candidatesNugetFullPaths = new List<string>();
	foreach (var customAnalyzerName in customAnalyzerNames)
	{
	var candidateAnalysisProjectPath = Path.Combine(TestAssetsRootPath, customAnalyzerName, $"{customAnalyzerName}.csproj");
	var nugetPackResults = RunnerUtilities.ExecBootstrapedMSBuild(
	$"{candidateAnalysisProjectPath} /m:1 -nr:False -restore /p:OutputPath={env.CreateFolder().Path} -getTargetResult:Build", out bool success, attachProcessId: false);
	success.ShouldBeTrue();

Versions & Configurations

MSBuild version 17.11.0-dev-24263-01+a8e224f80 for .NET
17.11.0.26301

macOS Sonoma 14.4.1
Apple M1 Pro (ARM)

macOS Ventura 13.6.6
Intel Core i7