getNumFrames sets Frame number from 0 to 1 before first frame finishes

[newville]

@TomQD-94

XSpress3::getNumFramesRead()

xspress3/xspress3App/src/xspress3Epics.cpp

Line 1967 in be84266

int Xspress3::getNumFramesRead()

sets the FrameCountParam from 0 to 1 well before the first frame is done.

Setting up to acquire 10 frames of 5 seconds each, and monitoring the ArrayCounter_RBV gives this:

~> camonitor 13QX7:det1:ArrayCounter_RBV 13QX7:det1:Acquire 13QX7:det1:ERASE
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:56:31.140269 0
13QX7:det1:Acquire             2025-06-17 10:55:42.676662 Done
13QX7:det1:ERASE               2025-06-17 10:56:31.140407 Erase
13QX7:det1:Acquire             2025-06-17 10:56:58.307833 Acquire
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:56:58.307956 1
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:08.307801 2
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:13.307802 3
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:18.307891 4
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:23.308066 5
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:28.308073 6
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:33.308064 7
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:38.308080 8
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:43.308110 9
13QX7:det1:ArrayCounter_RBV    2025-06-17 10:57:48.308194 10
13QX7:det1:Acquire             2025-06-17 10:57:48.308416 Done

Note that the counter goes to 1 as soon as Acquire starts, and is advanced to 2 after 10 seconds.

This comes from xsp3->scaler_check_progress. It appears that it returns 1 on the first call, but that subsequent calls before the first frame ends do return 0.

A possible workaround is to add a short delay before calling this in xsp3DataTaskC at

xspress3/xspress3App/src/xspress3Epics.cpp

Line 2036 in be84266

while (acquire && (frameNumber < numFrames)) {

to be something like (yes, it looks like it needs to be 30,000 microsceconds in my tests):

      while (acquire && (frameNumber < numFrames)) {
            if (acquired == 0) { 
                 usleep(30000);   \\ from unistd.h
            }
            acquired = pXspAD->getNumFramesRead();
            if (frameNumber < acquired) {

That seems kludgy. Any better ideas?

[TomQD-94]

Will look into this in more detail but to help me recreate the system.

• Is this using the Internal time frame generator?
• Array counter going to 1 immediatly suggests that the first frame will be shorter/blank is this the case? (I resolved a bug like this fairly recently)
• Do you get the same response in other trigger modes, i.e does acquire go to 1 immediately?

[newville]

@TomQD-94 Thanks. I should have said: this is in Internal Mode only. I don't see this when in TTL Veto mode. I don't think I am using the ITFG settings, but maybe I sure confirm that?

And, yes, the count time for Frame 1 is very short.

Which is unfortunate in "simple step scan", in Internal Mode, counting one frame. ;). I don't do that very often, but I think others do it all the time (so I'm not sure when this stopped working...).

A 25 to 30 msec delay in checking getNumFramesRead() seems to work, even in TTL veto mode. For example, if I set up 10-millisecond triggers, it appears from the clock times (I was testing with no X-rays) that points 1, 2, 3, 4, ... all seem to have the correct clock ticks. That may be because in Fly-Scanning, I always start the detector early (much more than 30 msec), and expect to throw out the first hot pixel.

[TomQD-94]

Hi Matt,

sorry took me a while to get round to this. I have some interesting results. Which i am unsure i am making a silly mistake on.

With the delay or without the delay the first frame I get as completely empty. This I have resolved this previously by removing the second call to histogram start (i will check this was the only change I made)

xspress3/xspress3App/src/xspress3Epics.cpp

Line 1240 in be84266

setupITFG();

All the other frames in my 10 * 5 sec acquisition were of the same length from clock ticks and the counts from playback data. This is with an X3X system.

[newville]

@TomQD-94 Thanks. Hmm, yes that section of code has always seemed very odd to me. The second setupITFG(); histogram_start() happens if the first call of histogram_start() gives an asynError. It seems odd to need to try twice and only report success if the first fails and the second succeeds. Why does the first call to histogram_start() fail?

I don't think I looked at that section for the behavior I was seeing. Could it be that calling settupITFG() a second time is the problem?

[TomQD-94]

@newville I have successfully confused myself, as that previous suggestion didn't fix anything.

I don't get the same behaviour as you are seeing. i.e a delay doesnt help with the first frame. What I see is that scaler_read_progress from within getNumFramesRead does return the expected result i.e 0 until the first frame is done. But I do see the numCaptured_RBV in the HDF5 plugin does increment immediately which triggers a frame to be processed/saved which is blank. If you run with one frame you can see in the gui that the data is there correctly.

It is possible to time the start of the scan to get a full first frame but this doesn't feel very robust. I would suspect that there is something sending a trigger to the HDF plugin too early. I have played around with when we read the frame but none of it helped with the HDF plugin triggering too early. I suspect that is where the issue is coming in.

Unless I have misunderstood your issue.

[newville]

@TomQD-94 The problem i was seeing was not when using the HDF5 plugin.
(in fact, I don't think I ever use the HDF5 plugin in Internal Mode ;)).

To reproduce, just put TriggerMode to Internal, set DwellTime to 1.0, NumImages to 10, and hit Acquire. ArrayCounter_RBV goes to 1 immediately, goes to 2 at 2 seconds, goes to 3 at 3 seconds, 4 at 4 seconds, etc. No plugins needed.

I think that scaler_check_progress() at

xspress3/xspress3App/src/xspress3Epics.cpp

Line 1967 in be84266

int Xspress3::getNumFramesRead()

is just returning 1 as a status indicator if it is called too early, even when the number of frames is 0. It might be that there is some part of the API to clear this, so that scaler_check_progress() will wait, but I don't know what that would be.

i'm going to merge this. If you have a better fix, please feel free to make an improved PR.

[BenBradnick]

Hi @newville I am unable to reproduce this on our Xspress 3 Mk2 system. I am running using our Mk2 branch at bb655ec and I commented out the original sleep you added to work around the problem:

if (acquired == 0) { usleep(25000); }

Does the frame number increment early for you in every acquisition? I tried several acquisitions with 0.1s, 1.0s and 5.0s frame times and the readback PV updated as expected, e.g.:

XSP3_8Chan:det1:Acquire        2025-09-11 10:01:21.953031 Acquire  
XSP3_8Chan:det1:ArrayCounter_RBV 2025-09-11 10:01:26.954046 1

The version of the Xspress libraries are slightly newer in this branch - do you want to try and see if you still get your issue when running it with your Xspress3 system without the sleep?

[newville]

@BenBradnick, I will try that. But also:

XSP3_8Chan:det1:Acquire        2025-09-11 10:01:21.953031 Acquire  
XSP3_8Chan:det1:ArrayCounter_RBV 2025-09-11 10:01:26.954046 1

Doesn't that mean the array counter goes to 1 almost immediately (within 0.001 seconds) after acquisition started?

That's the problem I am seeing and reporting here. If you set the acquisition time to 1 second, and ask for 10 frames, what do see? What I see is that frame 1 takes nearly 0 seconds, frame 2 takes 2 seconds, and frames 3 and higher takes 1 second.

[BenBradnick]

@newville there's a 5 second difference between those timestamps (21 versus 26 seconds). At 0.1s, 1.0s and 5.0s tests the timestamp for the readback PV updated as expected based on the frame time I set.

So it could be that is it because you are using an Xspress 3 system or because the versions of the library we are using is different. Unfortunately we don't have the older system at QD to test this with.

[newville]

Oh, okay, sorry. I see the 5 second delay now... it would be helpful to post monitors from multiple events to show that the time intervals are uniform.

I don't know if this is a difference in firmware or generation. To be clear, I am a customer with one system. But, are you not testing your firmware with multiple hardware generations?

I don't even know how to tell the version of the Xspress3 library.

Am I supposed to try some other Xspress3 library? If so, tell me exactly where that is. And show me version numbers and revision history for these.

[BenBradnick]

Hi @newville, apologies for not being clearer.

We do perform testing on different hardware configurations. Currently we have Xspress 3X Mk1 and Mk2 development systems. We no longer produce the original model that you have (Xspress 3, without the X) as it was superseded by the X models. Unfortunately there isn't an Xspress 3 system at QD we can use for testing.

I agree that the Xspress library is confusing in that the version number doesn't change. I will see if this can be fixed for future releases of the library as currently the version number is hard-coded to v1.0.0 and there is no revision history either (other than commit messages). This library is developed externally and not by us.

The PR #54 includes an update to the Xspress library - this was required for adding Mk2 support and is what I used for the testing above. I can see if we get the same behaviour as you on our Mk1 system and will report back here. It would be good to see if your Xspress 3 system's behaviour changes with this library - some parts of the library logic are common between the different generations of system.

I have also added #64 so that we can start tracking changes to this driver and backend library when things are changed. I am happy to summarise the changes in the Xspress library we this gets updated and will do this before the next release of this module.

[TomQD-94]

Hi @newville. I have just got round to testing this on the Mk1 system at QD. I see the same behaviour as on the Mk2 i.e the sleep is not required. Therefore, it seems this is only an issue on the X3 system or with the libraries that you are using.

XSP3_4Chan:det1:Acquire        2025-09-17 15:59:27.842687 Done  
XSP3_4Chan:det1:ArrayCounter_RBV 2025-09-17 16:01:44.574258 0  
XSP3_4Chan:det1:Acquire        2025-09-17 16:01:45.098843 Acquire  
XSP3_4Chan:det1:ArrayCounter_RBV 2025-09-17 16:01:47.099709 1  
XSP3_4Chan:det1:ArrayCounter_RBV 2025-09-17 16:01:49.099599 2  
XSP3_4Chan:det1:ArrayCounter_RBV 2025-09-17 16:01:51.099863 3  
XSP3_4Chan:det1:ArrayCounter_RBV 2025-09-17 16:01:53.099841 4  
XSP3_4Chan:det1:Acquire        2025-09-17 16:01:54.422532 Done

Would this allow the other pull request to be merged in.

I am starting to add to the documentation but would ideally create a seperate request for that once this is all tidied up.

@BenBradnick has also been looking into the other outstanding issues and trying to resolve as many as possible.
Cheers