Queued Sweep Bug?

I have a quick question about queued sweeps and how they work.

I have two signal hounds hooked up, and I basically just queue up 16 sweeps and then let them cook.

What’s happening is that sometimes it looks like I drop a sweep or am missing one or something, and then it re-inserts itself later.

The application very quickly switches between multiple RF sources, and does a scan and then switches to another one. For the sake of this effort, let’s say I have 20 different sources. I then plot a power graph showing the power at these different sources, and I continually do this over and over and over.

So I expect say a certain amount of power at source 10. And it’s usually correct. But sometimes that power shows up in the 9 slot, but then a couple of seconds later jumps back to the 10 slot. And then maybe 15 seconds later it jumps back to the 9 slot, and just kind of sometimes ping-pongs around but is only ever off by the single sweep.

The weird thing is that if I don’t queue sweeps (just use the normal smGetSweep), OR if I only queue 1 sweep it seems to work perfectly and all the energy stays in the bins it should be in. But anything above queueing a single sweep, and things get wonky.

Any thoughts?

Code outline is something like:

FastSpecAns[0].QueueSweeps(16);
FastSpecAns[1].QueueSweeps(16);
while (State == ArrayState.TRACKING)
{
SpectrumZero = FastSpecAns[0].GetFinishSweep();
SpectrumOne = FastSpecAns[0].GetFinishSweep();
}

And GetFinishSweep is basically just:

SmStatus f = sm_api.smFinishSweep(handle, QueuePos, null, Sweep, ref sweeptimeEpoch);
SmStatus s = sm_api.smStartSweep(handle, QueuePos);

QueuePos++;
if (QueuePos >= SweepsToQueue)
{
QueuePos = 0;
}

Kaiser,

Fun setup. There’s a lot going on here, so maybe I can start with some high level ideas and questions.

– How are you switching between your sources? Are you using the smSetSweepGPIO function?
– If yes to the question above, have you done any monitoring of those GPIO pins? Any way to verify they are cycling through all your states and not missing one periodically? Whether it’s hardware/software, that would be extremely helpful to know.
– Am I to understand that the “sweep offset” sticks around for many cycles? So if you are cycling through all 20 slots, are all 20 slots now off by 1 for some duration until it corrects itself? Or is it just slot 10 that shows up in slot 9 for several seconds until slot 9 reverts?
– Does using just 1 receiver make the problem go away? Even with 16 queued sweeps?
– Is it always a specific slot, e.g. 9/10? Or random?
– Does the “sweep offset” happen on both receivers? Or just one?
– Let’s say you’re switching through slots 8,9,10,11. Does it now look like 8,10,10,11 for 15 seconds and then go back to 8,9,10,11?

I’m trying to collect enough information that I could try to setup something here to reproduce, and narrow down the scope. I’m not aware of any outstanding issues with this functionality, but this is likely a more complex setup than most customers, and more complex than we’ve simulated here.

I look forward to your response.

[kaiser]

Andrew, thanks for the quick and thorough reply.

I’m going to keep investigating this some more myself to try and understand what’s going on and verify the rest of the system. It just *feels* like a Queued sweep bug since any time I go above 1 (basically queueing at all) it starts to happen. But feelings aren’t fact, so we’ll do what we can to finish narrowing down the issue.

– How are you switching between your sources? Are you using the smSetSweepGPIO function?

— No. We have the SM200’s set to produce a waveform (go low when scanning, transition high when near the end of the sweep) and that transition to high then tells the FPGA to move to the next position.

– If yes to the question above, have you done any monitoring of those GPIO pins?

— We monitor counters in the FPGA for that rising edge, and they always seem to line up…but then again, when you’re doing something at multiple kHz, and there’s only ever an off-by-one bug it would be a needle in a haystack.

We don’t think that we are missing triggers because otherwise it wouldn’t go back! We know if we have a bad trigger line because then what was in slot 10 goes to 9 and then to 8 and then to 7, and so on. But this corrects itself! Which is just odd.

– Any way to verify they are cycling through all your states and not missing one periodically? Whether it’s hardware/software, that would be extremely helpful to know.

— Yea, that is one of our concerns. I don’t know why it would go away when we set the queue to one, unless we were for example missing a trigger and then catching up somehow? But I don’t know how the FPGA would do that since we’re just looking for an edge transition on a pin. One of the avenues we are investigating on our end. But again, it only happens when we queue sweeps…one thought would be that the waveform / time the GPIO is high before the next sweep happens is longer when we’re not queuing, so we “catch” it more regularly…but then again, why would it correct itself!?! It would process moving bins in that instance, I would think.

– Am I to understand that the “sweep offset” sticks around for many cycles? So if you are cycling through all 20 slots, are all 20 slots now off by 1 for some duration until it corrects itself? Or is it just slot 10 that shows up in slot 9 for several seconds until slot 9 reverts?

— Yes, we really have a couple hundred “slots” that we cycle through. And we kind of see the whole thing shift one left, and hold there for a bit and then shift one right, back to the original and desired location. Then it’ll go back and forth doing that. Really odd, honestly.

– Does using just 1 receiver make the problem go away? Even with 16 queued sweeps?

— Our hardware and software architecture is very very intertwined with having two SM200’s working in concert with each other. I think we might be able to hack some places in the code to reduce it down to one signal hound, but only for a little bit before buffers and other things start overflowing and getting too far out of sync elsewhere. Otherwise it’d be a huge effort to get it down to just one.

– Is it always a specific slot, e.g. 9/10? Or random?

— It appears to be all the slots kind of shifting by one. Almost like we started getting scans out of order for a bit or something. It seems more like the problem would be elsewhere. But it is 100% solved when we queue only 1 scan, and is apparent when we go for more than one scan. I honestly haven’t been able to come up with a theory as to why / how the queued/non-queued is at play here. But as far as we can tell, it is.

– Does the “sweep offset” happen on both receivers? Or just one?

— It does happen to both. We are looking into that more this rest of this week. We had an issue preventing us from looking at both and seeing if they do it at the same time, or if it’s a random correlation between the two.

– Let’s say you’re switching through slots 8,9,10,11. Does it now look like 8,10,10,11 for 15 seconds and then go back to 8,9,10,11?

It looks like 1,2,3,4,5,6,7,8,9 goes to 2,3,4,5,6,7,8,? for a while and then back to 1->9 in order. We’re investigating whether the ? is just old data in there, or if it’s 1 from the previous slot. It’s actually fairly hard for us to put energy *only* in one slot with our current setup, but figuring this out I think might provide a lot of insight, and that’s what we are attempting today.

It’s a tough nut to crack, because the entire system is orchestrating this at multiple kHz, and there’s a lot at play here, and we’ve tried a ton of things across all the other components, both hardware and software at play here, and it wasn’t until we only queued 1 sweep up did anything fix it.

With queued sweeps, is there maybe some extra allocation / deallocation (and thus garbage collection or other process) going on under the covers in the driver that’s maybe going on here?

• This reply was modified 7 months, 3 weeks ago by kaiser.

Kaiser,

This is good information, thanks for the thorough reply.

What are typical sweep settings you are using? (span/rbw/etc). If I try to reproduce I want to get as close as possible to your setup. Thankfully we are about to release a high speed GPIO controlled switch, so it shouldn’t be too hard to simulate this!

Thanks Andrew.

I do have to say that with SM200C’s, the difference in speed between Queued Sweeps and non-queued is basically imperceptible in our application. It’s very very response non-queued…I’ll be able to test how the SM200B’s hold up doing the same. If I remember right the non-queued was significantly slower.

Here’s some of the various code bits for how we’re configuring the SignalHound.

sm_api.smSetSweepSpeed(handle, SmSweepSpeed.smSweepSpeedFast);
CurrentStatus = sm_api.smSetSweepCoupling(handle, 125.0e3, 125.0e3, 0.0001); //Gives a 3.1kHz scan, 138us blanking (similar spectrum sweep time, about 50% duty cycle).
sm_api.smSetSweepDetector(handle, SmDetector.smDetectorAverage, SmVideoUnits.smVideoLog);

sm_api.smSetSweepWindow(handle, SmWindowType.smWindowNutall);
sm_api.smSetSweepSpurReject(handle, SmBool.smFalse);
sm_api.smSetPreselector(handle, SmBool.smFalse);

//GPIO configuration
int gpiosteps = 2;
sm_api.SmGPIOStep[] Steps = new sm_api.SmGPIOStep[gpiosteps];

Steps[0].freq = 2.2e9;
Steps[1].freq = 2.38e9;
Steps[0].mask = 0x00;
Steps[1].mask = 0x0F;

sm_api.smSetGPIOState(handle, SmGPIOState.smGPIOStateOutput, SmGPIOState.smGPIOStateOutput);

[Justin Crooks]

Kaiser,
I’m curious about the coordinating of the two SM’s. Are they both producing a GPIO output that the FPGA is using to coordinate the signal generation? Do you monitor the two GPIO outputs continuously? Does the bug follow e.g. SM #2 trigger arriving before SM #1? Or SM #2 getting a full sweep ahead or behind SM #1?
Our sweeps are not deterministic, so I can certainly see some variance in timing between the two SM’s playing a role in this.
If you could use something like I/Q streaming, and have the FPGA generate the trigger, or have SM #1 sweeping and generating the trigger and SM #2 streaming at one of the frequencies with a trigger input, something like this might be more solid. Just my thoughts…

• This reply was modified 7 months, 3 weeks ago by Justin Crooks.

We have the two SM200’s together on the same 10MHz reference (one outputs it, and the other inputs it), and it seems to keep them well sync’d.

Two separate triggers go into the FGPA from the SignalHounds, and cause two separate things to happen. Now, while the FPGA is off doing one thing for the first trigger, theoretically the second trigger could get missed. But we’ve timed the actions on the O-scope, and the FPGA goes back to listening in under half the time that the signalhound keeps the GPIO high while resetting. So we don’t think the issue is there (although this is one area for us to revisit if the rest don’t turn anything up). But again, if we were missing a trigger, we’d expect to never be able to recover from that miss — that we’d always be off by one…so it’s odd.

Right now we cover ~200MHz of spectrum, so IQ streaming isn’t feasible, especially since we do have a lot of SM200B’s in our legacy and other designs…but we’ve definitely been toying and playing around with it some. It’s nice to have the sweep handed to us with all the FFTs and windowing and everything done, instead of having to do that ourselves. But one day we might bite the bullet and get there (and probably will need to soon for another project we’re using the SM2000’s on).

I wanted to come back to this — sorry, I had some work travel and haven’t been able to finish off updating y’all and doing some testing.

What we found:

The SM200B’s *did not* have this synchronization issue. When we kick them both off at the same time, with the same 10MHz reference and queue scans, it can run for hours and still be 1:1 in sync with scans.

With the SM200C’s, we notice a divergence right away. I assume with the latency inherent in networking, we end up with multiple scans on one done before the other even gets the command. Makes some sense. But I would’ve thought that if we queued the same # of sweeps and finished them that we’d see identical numbers at the end, which we aren’t. It might be an issue with our trigger line for counting sweeps?

So we went to try and work around it, but just doing the single sweeps at a time. Which works for us, and seems to count correctly (potentially exonerating our trigger line? If it counts correctly here, then it should be able to count queued sweeps too, right? The queued sweeps are ~20% faster though…but that’s fine for us for now).

I think that we might need to scope the GPIO outputs during these sweeps to figure out what’s going on.

Thanks for the update Kaiser. I have not set up this configuration yet to do any testing. I will use SM200C’s when I do get to this.

[Author]

Posts