Some confusions about AUCM loss

[ghost]

Dear developers,

After learning about your work, I have the following confusions:

1. Why do we need to convert the square loss into an SPP problem? What problems will arise if we directly minimize $E[f(x_i)-a]^2+E[f(x_j)-b]^2+2*(m+b-a)^2$?

loss = self.mean((y_pred - self.a)**2*pos_mask) + self.mean((y_pred - self.b)**2*neg_mask) + (self.margin + self.mean(y_pred*neg_mask) - self.mean(y_pred*pos_mask))**2

2. I noticed that in your code, $a$ in $E[f(x_i)-a]^2$ and $b$ in $E[f(x_j)-b]^2$ directly use self.a and self.b, while $a$ and $b$ in $(m+b-a)$ use the sample mean self.mean(y_pred*neg_mask) - self.mean(y_pred*pos_mask). I would like to know the reason for that.

loss = self.mean((y_pred - self.a)**2*pos_mask) + self.mean((y_pred - self.b)**2*neg_mask) + \ 2*self.alpha*(self.margin + self.mean(y_pred*neg_mask) - self.mean(y_pred*pos_mask)) - self.alpha**2

3. Can I regard that the design of margin loss is to transform the square loss $(m+f(x_j)-f(x_i))^2$ into $max[0, (m+f(x_j)-f(x_i))]^2$ to allow $m+f(x_j)$ to be equal or less than $f(x_i)$ while the square loss only seeks to be equal to $f(x_i)$? When the loss function has a value of 0, is there a potential problem that the gradient cannot be updated?

4. In the demo you provided, the AUC test score of AUCM based on PESG on CIFAR10 can reach 0.9245, while the value quoted in your paper Large-scale Robust Deep AUC Maximization is 0.715±0.008. Is this because some content has been updated?

I would be grateful if you could reply as soon as possible. Wish you a happy new year.

[optmai]

Thank you for your interest in our library. Let me answer your questions below.

1. You cannot simply do that because it does not provide guarantee for convergence. Because we use mini-batch to estimate the gradient, the loss you defined here loss = self.mean((y_pred - self.a)**2pos_mask) + self.mean((y_pred - self.b)**2neg_mask) + (self.margin + self.mean(y_predneg_mask) - self.mean(y_predpos_mask))**2 will only use the data in the mini-batch. The gradient of this mini-batch loss is not an unbiased estimator for the true gradient. Converting this into a SPP will help address this issue. Additionally, the SPP allows more efficient optimization in online learning and distributed learning.

2. Indeed, a, b are mean scores of positive and negative samples. We make them free variable in the first two terms so that their optimal solution are indeed the mean scores. We cannot make a, b as free variables in the last term because the resulting solution will not be the mean scores any more.

3. I do not think you can explain in that way. It is not equivalent to the SPP.

4. I think it is because that we make the data more imbalanced in the paper.

Please let us know if you have any additional questions.

[ghost]

Thanks for your reply.

As you said, the gradient of the mini-batch loss is not an unbiased estimator for the true gradient, and converting this into a SPP will help address this issue. Is there any theoretical support of that. I mean why the gradient of mini-batch loss is not an unbiased estimator? And how SPP works here? In addition, it has been proved that the square loss is convergent in paper One-Pass AUC Optimization without conversion to SPP.

And in the last term, the $m+E(f(x_j))-E(f(x_i))$, don't $E(f(x_j))$ and $E(f(x_i))$ mean the scores of positive and negative samples?

Thanks for your patient reply again.

[optmai]

Hi Rickey, I am sorry for the late reply. The reason is this: the third term in the objective is (E f(n) - E f(p) + c)^2 where E denote the expectation, n is random negative data and p is a random positive data, c is a constant margin. When you calculate the gradient, you have (E f(n) - E f(p) + c) *\nabla (E f(n) - E f(p) + c). You need the minibatch to estimate the expectation. But if you use the same batch, you have a biased estimator. You can address this by using different batch for the two expectations (before and after the *). But this reduce the effective batch size. If we reformulate it into max, we have max_alpha 2*alpha *(E f(n) - E f(p) + c) - alpha^2, then you will not have this issue any more as you will update alpha using its unbiased gradient estimator and the update the w same way. You may want to check this paper https://arxiv.org/abs/2202.12396. See below for answer to the other question. Regards Tianbao On Jan 7, 2025, at 10:05 PM, Rickey ***@***.***> wrote:  Thanks for your reply. As you said, the gradient of the mini-batch loss is not an unbiased estimator for the true gradient, and converting this into a SPP will help address this issue. Is there any theoretical support of that. I mean why the ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for your reply. As you said, the gradient of the mini-batch loss is not an unbiased estimator for the true gradient, and converting this into a SPP will help address this issue. Is there any theoretical support of that. I mean why the gradient of mini-batch loss is not an unbiased estimator? And how SPP works here? And in the last term, the $m+E(f(x_j))-E(f(x_i))$, don't $E(f(x_j))$ and $E(f(x_i))$ mean the scores of positive and negative samples? This is correct. Thanks for your patient reply again. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/Optimization-AI/LibAUC/issues/67*issuecomment-2576703447__;Iw!!KwNVnqRv!Ew01fFMG9E7oxqid3YmiZwSkMfQSlP11c3OwdxshxGmCk9B4JzGqVrAE0yWHr-09Cwz2pcOkFoyZFIV1BFZSlVbTkgWi$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BAF3BBCZL4SA7HCJL2P5HU32JSPZTAVCNFSM6AAAAABURJNTLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZWG4YDGNBUG4__;!!KwNVnqRv!Ew01fFMG9E7oxqid3YmiZwSkMfQSlP11c3OwdxshxGmCk9B4JzGqVrAE0yWHr-09Cwz2pcOkFoyZFIV1BFZSlR3xEylA$>. You are receiving this because you modified the open/close state.Message ID: ***@***.***>