Timeouts in `System.Text.Json.Tests` on arm32

[AaronRobinsonMSFT]

#67858

Console log: 'System.Text.Json.Tests' from job 0e2f0129-97a5-4449-a6f1-4f6d48f286d1 (ubuntu.1804.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 on ddvsotx2l053
running $HELIX_CORRELATION_PAYLOAD/scripts/437cee12e49841d097023962d57d444e/execute.sh in /ssd/work/A22408F2/w/A0840945/e max 2700 seconds

=================

Console log: 'System.Text.Json.Tests' from job cc08232a-0565-4446-a309-8e67f6b5ff0a (ubuntu.1804.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 on ddvsotx2l143
running $HELIX_CORRELATION_PAYLOAD/scripts/00f3104607ef4bc4ab7713de646b08be/execute.sh in /ssd/work/A07A0914/w/B302093F/e max 2700 seconds

#67852

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

#67858

Console log: 'System.Text.Json.Tests' from job 0e2f0129-97a5-4449-a6f1-4f6d48f286d1 (ubuntu.1804.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 on ddvsotx2l053
running $HELIX_CORRELATION_PAYLOAD/scripts/437cee12e49841d097023962d57d444e/execute.sh in /ssd/work/A22408F2/w/A0840945/e max 2700 seconds

=================

Console log: 'System.Text.Json.Tests' from job cc08232a-0565-4446-a309-8e67f6b5ff0a (ubuntu.1804.armarch.open) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 on ddvsotx2l143
running $HELIX_CORRELATION_PAYLOAD/scripts/00f3104607ef4bc4ab7713de646b08be/execute.sh in /ssd/work/A07A0914/w/B302093F/e max 2700 seconds

#67852

Author:	AaronRobinsonMSFT
Assignees:	-
Labels:	arch-arm32, area-System.Text.Json, os-linux
Milestone:	-

[jkoritzinsky]

Happening here as well: #67676 (comment)

[elinor-fung]

Hit again in #67635.

A bunch of different tests being detected as long running - from one of the logs:

[Long Running Test] 'System.Text.Json.Serialization.Tests.ConstructorTests_AsyncStream.MultipleThreads', Elapsed: 00:30:23
[Long Running Test] 'System.Text.Json.Serialization.Tests.ReferenceHandlerTestsDynamic_AsyncStream.TestJsonPathDoesNotFailOnMultiThreads', Elapsed: 00:30:42
[Long Running Test] 'System.Text.Json.Serialization.Tests.ReferenceHandlerTestsDynamic_AsyncStreamWithSmallBuffer.TestJsonPathDoesNotFailOnMultiThreads', Elapsed: 00:29:31
[Long Running Test] 'System.Text.Json.Serialization.Tests.StreamTests_Sync.VeryLargeJsonFileTest', Elapsed: 00:24:47
[Long Running Test] 'System.Text.Json.Serialization.Tests.CacheTests.MultipleTypes', Elapsed: 00:28:42
[Long Running Test] 'System.Text.Json.Serialization.Tests.ConstructorTests_SyncStream.MultipleThreads', Elapsed: 00:30:17
[Long Running Test] 'System.Text.Json.Serialization.Tests.ConstructorTests_Span.MultipleThreads', Elapsed: 00:29:08
[Long Running Test] 'System.Text.Json.Serialization.Tests.CollectionTestsDynamic_AsyncStream.WriteNestedAsyncEnumerable', Elapsed: 00:25:21
[Long Running Test] 'System.Text.Json.Serialization.Tests.CollectionTestsDynamic_AsyncStreamWithSmallBuffer.WriteSequentialNestedAsyncEnumerables', Elapsed: 00:28:11
[Long Running Test] 'System.Text.Json.Serialization.Tests.ConstructorTests_String.MultipleThreads', Elapsed: 00:26:51
[Long Running Test] 'System.Text.Json.Serialization.Tests.StreamTests_Async.VeryLargeJsonFileTest', Elapsed: 00:09:22
[Long Running Test] 'System.Text.Json.Tests.JsonDocumentTests.ParseJson_UnseekableStream', Elapsed: 00:26:40
[Long Running Test] 'System.Text.Json.Serialization.Tests.ReferenceHandlerTestsDynamic_String.TestJsonPathDoesNotFailOnMultiThreads', Elapsed: 00:26:39
[Long Running Test] 'System.Text.Json.Serialization.Tests.StreamTests_DeserializeAsyncEnumerable.DeserializeAsyncEnumerable_ReadSimpleObjectAsync', Elapsed: 00:25:34
[Long Running Test] 'System.Text.Json.Serialization.Tests.StreamTests_AsyncWithSmallBuffer.VeryLargeJsonFileTest', Elapsed: 00:06:04

[elinor-fung]

There over 50 hits in the past day - marking as blocking-clean-ci. Nothing in the week before that though - some recent change?

WorkItems
| where FriendlyName == "System.Text.Json.Tests"
| where  Queued > ago(1d)
| where Status == "Timeout"
| where QueueName contains "arm"
| join Jobs on JobId
| project
  Queued,
  FriendlyName, ExitCode, Status,
  ConsoleUri,
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  RuntimeFlavor = tostring(parse_json(Properties).runtimeFlavor),
  BuildId = tostring(parse_json(Properties).BuildId),
  QueueName, Source