Fix test criteria to fix paqa_latency.c by philburk · Pull Request #1111 · PortAudio/portaudio

philburk · 2025-12-26T21:36:38Z

It was failing because the latency was too low.

Now we allow lower latency.

Fixes #742

philburk · 2025-12-26T23:49:48Z

After discussion with Ross, the plan is to:

Start checking at 0.0
expect the finalLatency to be >= suggested, UNTIL it clamps and then it should not increase.

philburk · 2026-01-03T01:26:46Z

expect the finalLatency to be >= suggested, UNTIL it clamps and then it should not increase.

I added that test. But we are seeing failures where the latency hits the maximum and then keeps going up!

Using device #4: 'Maono PD200W Mic USB' (Core Audio)
------------------------ paqaCheckMultipleSuggested - INPUT
 lowLatency  = 0.00441667
 highLatency = 0.01375
 numChannels = 1
 sampleRate  = 48000
   suggestedLatency[ 0] = 0.000000, finalLatency = 0.003396
   suggestedLatency[ 1] = 0.001375, finalLatency = 0.003396
   suggestedLatency[ 2] = 0.002750, finalLatency = 0.003396
   suggestedLatency[ 3] = 0.004125, finalLatency = 0.004125
   suggestedLatency[ 4] = 0.005500, finalLatency = 0.005500
   suggestedLatency[ 5] = 0.006875, finalLatency = 0.006875
     maximumLatency = 0.006875
   suggestedLatency[ 6] = 0.008250, finalLatency = 0.008250
/Users/phil/Work/portaudio/pagit/qa/paqa_latency.c:310 - ERROR - Latency should be == maximumLatency
INPUT CHECK FAILED !!! #4: 'Maono PD200W Mic USB'

also

Using device #7: 'ZoomAudioDevice' (Core Audio)
------------------------ paqaCheckMultipleSuggested - OUTPUT
 lowLatency  = 0.0333333
 highLatency = 0.0426667
 numChannels = 2
 sampleRate  = 48000
   suggestedLatency[ 0] = 0.000000, finalLatency = 0.032313
   suggestedLatency[ 1] = 0.004267, finalLatency = 0.032313
   suggestedLatency[ 2] = 0.008533, finalLatency = 0.032313
   suggestedLatency[ 3] = 0.012800, finalLatency = 0.032313
   suggestedLatency[ 4] = 0.017067, finalLatency = 0.032313
   suggestedLatency[ 5] = 0.021333, finalLatency = 0.032313
   suggestedLatency[ 6] = 0.025600, finalLatency = 0.032313
   suggestedLatency[ 7] = 0.029867, finalLatency = 0.032313
   suggestedLatency[ 8] = 0.034133, finalLatency = 0.034125
     maximumLatency = 0.034125
   suggestedLatency[ 9] = 0.038400, finalLatency = 0.038396
/Users/phil/Work/portaudio/pagit/qa/paqa_latency.c:310 - ERROR - Latency should be == maximumLatency
OUTPUT CHECK FAILED !!! #7: 'ZoomAudioDevice'

RossBencina · 2026-02-06T00:31:07Z

NEXT STEP: confirm that rounding error is the problem. investigate source of rounding error. consider revising the spec

I think that the test is valid with respect to the current spec.

I think the failing tests reflect a bug in the code in the sense that the test results indicate a violation of the current spec for suggestedLatency in portaudio.h here:

portaudio/include/portaudio.h

Lines 570 to 580 in 5d01510

    
               /** The desired latency in seconds. Where practical, implementations should 
        
                configure their latency based on these parameters. Implementations should 
        
                round the actual latency up to the next viable value, except when suggested 
        
                latency exceeds the upper limit for the device. 
        
                Actual latency values for an open stream may be retrieved using the 
        
                inputLatency and outputLatency fields of the PaStreamInfo structure 
        
                returned by Pa_GetStreamInfo(). 
        
                @see default*Latency in PaDeviceInfo, *Latency in PaStreamInfo 
        
               */ 
        
               PaTime suggestedLatency;

The USB mic result is difficult to interpret because the displayed floating point precision doesn't show that actual latency < suggested latency, even though it must be by the logic of the test.

I suspect the issue is that there is rounding down when the spec says to always round up. This could easily be a bug in PA/CoreAudio. It would be good to know whether the test also fails with other host APIs.

We are undecided whether rounding down within some epsilon should be in specification . It would be user friendly to allow for +/- half (or 1/4 or 0.1) sample period of slop to allow for incorrect rounding in floating point calculations.

I definitely think that the actual latency should be reported as accurately as possible, so the code should not "pretend" to have honoured the suggested latency when it actually rounded down, even a little bit.

RossBencina · 2026-02-06T00:35:28Z

Suggestions for improving the test:

Rename maximumLatency -> detectedMaximumLatency and reflect this in the test output/printfs
Add a third column "final - suggested" where we expect that column to be >= 0. A negative value indicates that the value was rounded down and will indicate by how much.
Could print the sample period as a reference value. eg display "sample period: 0.000020" under "sampleRate = 44100" where sample period is 1.0/sampleRate

philburk · 2026-02-19T23:23:05Z

Print the delta value between final and suggested latency with scientific notation.

RossBencina

Phil and I agreed to merge this as a failing test once the following two improvements are made:

Print in seconds (next to sample rate) 1/sampleRate
Print suggested, final delta

Just expect monotonic increases, and not much higher latency than suggested. Fixes #742

philburk · 2026-07-16T23:01:13Z

I made the suggested changes. Now I get this:

Using device #4: 'Maono PD200W Mic USB' (Core Audio)
------------------------ paqaCheckMultipleSuggested - INPUT
 lowLatency   = 0.00441667
 highLatency  = 0.01375
 numChannels  = 1
 sampleRate   = 48000 Hz
 samplePeriod = 2.083333e-05 seconds
   suggestedLatency[ 0] = 0.000000, finalLatency = 0.003396, (final - suggested) = 3.395833e-03
   suggestedLatency[ 1] = 0.001375, finalLatency = 0.003396, (final - suggested) = 2.020833e-03
   suggestedLatency[ 2] = 0.002750, finalLatency = 0.003396, (final - suggested) = 6.458333e-04
   suggestedLatency[ 3] = 0.004125, finalLatency = 0.004125, (final - suggested) = 0.000000e+00
   suggestedLatency[ 4] = 0.005500, finalLatency = 0.005500, (final - suggested) = 0.000000e+00
   suggestedLatency[ 5] = 0.006875, finalLatency = 0.006875, (final - suggested) = -8.673617e-19
     detectedMaximumLatency = 0.006875
   suggestedLatency[ 6] = 0.008250, finalLatency = 0.008250, (final - suggested) = 0.000000e+00
/Users/phil/Work/portaudio/pagit/qa/paqa_latency.c:312 - ERROR - Latency should be == detectedMaximumLatency
INPUT CHECK FAILED !!! #4: 'Maono PD200W Mic USB'

See attachment for the full log: qa_latency_PR1111_20160716.txt

philburk · 2026-08-08T00:41:03Z

TODO
Show the level of nesting for the tests. Indentation.
Show delta in number of sample frames.

Could add a test for more serious failures, "smoke".
Separate out the tests so one known failure cannot mask new or more serious errors.

Call this monotonic test paqa_latency_monotonic.c

Suggest 0.0, get >= defaultMinLatency.
Check final >= 0.0.

RossBencina · 2026-08-08T00:48:18Z

I think test could be merged now, but I agree that making the output easier to interpret would be helpful.

We also discussed splitting out a smoke test:

suggested: 0 -> final > 0 && <= default-min
suggested: default min -> final >= default min
suggested: default max -> final >= default max

This allows the rest of paqa_latency.c to pass.

philburk · 2026-08-14T00:44:25Z

I pulled out the monotonic test that was failing.
Now it runs more tests that came after the failing test.
I also improved the appearance of the test output.

philburk · 2026-08-14T00:44:56Z

@@ -0,0 +1,244 @@
+/** @file paqa_latency.c


Fix file name.

philburk · 2026-08-14T00:45:24Z

@@ -0,0 +1,244 @@
+/** @file paqa_latency.c
+    @ingroup qa_src
+    @brief Test latency estimates.


Describe monotonic test.

philburk · 2026-08-14T00:45:42Z

+    @author Phil Burk <philburk@softsynth.com>
+*/
+/*
+ * $Id: patest_sine.c 1368 2008-03-01 00:38:27Z rossb $


Fix test name.

philburk · 2026-08-14T00:47:30Z

I need to run autoconf to add the new test.

RossBencina

Looks good. I've made some suggestions to improve readability.

RossBencina · 2026-08-14T01:04:35Z

+
+    if( highLatency < 0.001 )
+    {
+        numLoops = 2; /* Just test 0 and high. Don't divide by 0. */


Comment is a bit cryptic. It might be clearer if you assign both values of numLoops here, something like:

numLoops = (highLatency < 0.001) ? 2 /* test suggestedLatency 0 and highLatency. note numLoops must be > 0 to avoid divzero */ : 11;

Buy why 11?

RossBencina · 2026-08-14T01:14:05Z

+    PaStreamParameters streamParameters;
+    const PaStreamInfo* streamInfo;
+    double lowLatency;
+    double highLatency;


I'm not a fan of "final". It doesn't carry enough meaning, especially given that is exactly the quantity being tested by this test. Suggest rename finalLatency to reportedLatency or to streamInfoLatency, I have a slight preference for reportedLatency but if you want to keep it short could go with 'infoLatency' or 'streamLatency'. I think we already agreed that "actual latency" is not the right concept here.

RossBencina · 2026-08-14T01:15:02Z

+static int paqaCheckMultipleSuggested( PaDeviceIndex deviceIndex, int isInput )
+{
+    int i;
+    int numLoops = 11;


Why 11? move initialization above loop see comment below.

RossBencina · 2026-08-14T01:16:09Z

+        printf(INDENT "suggested[%2d] = %8.6f", i, streamParameters.suggestedLatency );
+
+        err = Pa_OpenStream(
+                            &stream,


check style guide for indent policy. i think 2 tabstops right of block

RossBencina · 2026-08-14T01:23:49Z

+
+#define INDENT "  "
+/*******************************************************************/
+static int paqaCheckMultipleSuggested( PaDeviceIndex deviceIndex, int isInput )


?rename paqaCheckStreamLatencyIsMonotonic or paqaCheckStreamMonotonicLatency

RossBencina · 2026-08-14T01:52:51Z

+        /* If we are not at maximum then we should be rounding up. */
+        if (atMaximumLatency == 0) {
+            QA_ASSERT_TRUE("Latency should be >= suggestedLatency",
+                           finalLatency >= streamParameters.suggestedLatency);
+        } else {
+            QA_ASSERT_TRUE("Latency should be == detectedMaximumLatency",
+                           finalLatency == detectedMaximumLatency);
+        }


Suggested change

/* If we are not at maximum then we should be rounding up. */

if (atMaximumLatency == 0) {

QA_ASSERT_TRUE("Latency should be >= suggestedLatency",

finalLatency >= streamParameters.suggestedLatency);

} else {

QA_ASSERT_TRUE("Latency should be == detectedMaximumLatency",

finalLatency == detectedMaximumLatency);

}

if (atMaximumLatency == 0) {

/* Below the maximum, stream should always round latency up. */

QA_ASSERT_TRUE("Latency should be >= suggestedLatency",

finalLatency >= streamParameters.suggestedLatency);

} else {

/* Once the maximum has been reached, reported latency should remain constant. */

QA_ASSERT_TRUE("Latency should be == detectedMaximumLatency",

finalLatency == detectedMaximumLatency);

}

RossBencina · 2026-08-14T01:55:25Z

+        // Get the latency from the streamInfo now because it will be invalid after the
+        // stream is closed.


C++ comments (seems weird having a mixture in this file)

RossBencina · 2026-08-14T02:04:52Z

+                        finalLatency >= previousLatency);
+        QA_ASSERT_TRUE("Latency should be > 0.0", finalLatency > 0.0);
+        if (atMaximumLatency == 0) {
+            /* If we get a lower value then we must be clipping at max latency. */


Suggested change

/* If we get a lower value then we must be clipping at max latency. */

/* When not yet at the maximum, interpret a reported stream latency

that is less than the suggested latency as clipping at max. */

RossBencina · 2026-08-14T02:12:47Z

+        QA_ASSERT_TRUE("Latency should be monotonically non-decreasing with suggested latency.",
+                        finalLatency >= previousLatency);
+        QA_ASSERT_TRUE("Latency should be > 0.0", finalLatency > 0.0);


Put sanity check first.

Suggested change

QA_ASSERT_TRUE("Latency should be monotonically non-decreasing with suggested latency.",

finalLatency >= previousLatency);

QA_ASSERT_TRUE("Latency should be > 0.0", finalLatency > 0.0);

QA_ASSERT_TRUE("Latency should be > 0.0", finalLatency > 0.0);

QA_ASSERT_TRUE("Latency should be monotonically non-decreasing with increasing suggested latency.",

finalLatency >= previousLatency);

RossBencina · 2026-08-14T03:26:16Z

+        QA_ASSERT_TRUE("Latency should be monotonically non-decreasing with suggested latency.",
+                        finalLatency >= previousLatency);
+        QA_ASSERT_TRUE("Latency should be > 0.0", finalLatency > 0.0);
+        if (atMaximumLatency == 0) {


Add explanatory comment:

Suggested change

if (atMaximumLatency == 0) {

/* The state machine below checks that the reported stream latency

* remains at or above the suggested latency until the point at which

* the reported stream latency clamps at its maximum.

*

* latency ^ .

* value | ___._______

* | /.

* | _- '. . . . suggested latency

* | / . _______ reported ("final") latency

* | -.

* '--------------------->

* 0 loop iteration

*/

if (atMaximumLatency == 0) {

philburk added this to the V19.8 milestone Dec 26, 2025

philburk self-assigned this Dec 26, 2025

philburk requested a review from RossBencina December 26, 2025 21:36

philburk assigned RossBencina Dec 26, 2025

RossBencina reviewed Dec 26, 2025

View reviewed changes

Comment thread qa/paqa_latency.c Outdated

philburk force-pushed the latency_qa_742 branch from deb402c to fbecb61 Compare January 3, 2026 01:19

RossBencina requested changes Feb 19, 2026

View reviewed changes

RossBencina added the test-qa Test code in /qa (automated test program) label Mar 27, 2026

philburk added 3 commits July 16, 2026 15:17

Change test criteria so that lower latency is OK

688418f

Just expect monotonic increases, and not much higher latency than suggested. Fixes #742

Fix alignment for continuation lines

faeb2a4

Check to make sure latency does not drop below maximumLatency

3093947

print final-suggested and samplePeriod

fe6d39e

philburk force-pushed the latency_qa_742 branch from fbecb61 to fe6d39e Compare July 16, 2026 23:03

separate out paqa_latency_monotonic.c, which is failing

b4cd84c

This allows the rest of paqa_latency.c to pass.

philburk commented Aug 14, 2026

View reviewed changes

RossBencina reviewed Aug 14, 2026

View reviewed changes

		// Get the latency from the streamInfo now because it will be invalid after the
		// stream is closed.

	/* If we get a lower value then we must be clipping at max latency. */
	/* When not yet at the maximum, interpret a reported stream latency
	that is less than the suggested latency as clipping at max. */

-        if (atMaximumLatency == 0) {
+        /* The state machine below checks that the reported stream latency
+ 	 * remains at or above the suggested latency until the point at which
+	 * the reported stream latency clamps at its maximum.
+	 *
+	 * latency ^                 .
+	 *   value |           ___._______
+	 *         |          /.
+	 *         |    _- '.                 .  .  .  suggested latency
+	 *         |   / .                    _______  reported ("final") latency
+	 *         | -.
+	 *         '--------------------->
+	 *        0         loop iteration
+	 */
+        if (atMaximumLatency == 0) {

Uh oh!

Conversation

philburk commented Dec 26, 2025

Uh oh!

Uh oh!

philburk commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philburk commented Jan 3, 2026

Uh oh!

RossBencina commented Feb 6, 2026

Uh oh!

RossBencina commented Feb 6, 2026

Uh oh!

philburk commented Feb 19, 2026

Uh oh!

RossBencina left a comment

Choose a reason for hiding this comment

Uh oh!

philburk commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philburk commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RossBencina commented Aug 8, 2026

Uh oh!

philburk commented Aug 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philburk commented Aug 14, 2026

Uh oh!

RossBencina left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RossBencina Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

philburk commented Dec 26, 2025 •

edited

Loading

philburk commented Jul 16, 2026 •

edited

Loading

philburk commented Aug 8, 2026 •

edited

Loading

RossBencina Aug 14, 2026 •

edited

Loading