The very point of a question like this is to test your ability to not freak out at
the sight of something new and unfamiliar. You’re not expected to have any
prior experience with this operation: The test makers just made it up! But they
also defined it for you, so as long as you don’t freak out, everything you need
to answer the question is right there.
The operation § is defined as and when a = −3 and b = x, the
result is 0.
The answer is (A).
DON’T FREAK OUT
If you see something—a symbol, a word—you’ve never seen before,
chances are the test makers just made it up to test your ability to stay
cool in the face of something new and unfamiliar.