Skip to content

Commit 7d93938

Browse files
hosseinfanihosseinfani
authored
Merge pull request fani-lab#314 from mahdis-saeedi/main
(fani-lab#312) - argpartition does not garantee the sorting of the top-k return indexes. it only garantees that the top-k return is the top-k among all items. No issue as the pytrec_eval does the sorting itself but needs correct scores. - pytrec_eval does not work with numpy.float. So, the casting to built-in float is required.
2 parents 3c7ca1e + 9315c83 commit 7d93938

File tree

7 files changed

+99
-1
lines changed

7 files changed

+99
-1
lines changed

‎docs/e2e_journal_experiment/cpu_gpu_usage/bnn_merge_teams_w_same_skills=False.log‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
[2025-07-21 11:20:10,240][cmn.team][INFO] - Loading teamsvecs matrices from /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/teamsvecs.pkl ...
2+
[2025-07-21 11:20:10,728][cmn.team][INFO] - Loading indexes pickle from /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/indexes.pkl ...
3+
[2025-07-21 11:20:10,790][cmn.team][INFO] - Indexes pickle is loaded.
4+
[2025-07-21 11:20:10,790][cmn.team][INFO] - Teamsvecs matrices and indexes for skills (186385, 27), members (186385, 44774), and locations None are loaded.
5+
[2025-07-21 11:20:10,792][__main__][INFO] - Loading splits from /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/splits.pkl ...
6+
[2025-07-21 11:20:10,795][cmn.team][INFO] - Loading member-skill co-occurrence matrix (44774, 27) from /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/skillcoverage.pkl ...
7+
[2025-07-21 11:20:17,314][__main__][INFO] - Training team recommender instance mdl.bnn.Bnn ... 
8+
[2025-07-21 11:21:00,479][mdl.fnn][INFO] - Fold 0/4, Epoch 0, Train Loss: 188144.8602
9+
[2025-07-21 11:21:00,480][mdl.fnn][INFO] - Fold 0/4, Epoch 0, Valid Loss: 117498.1362
10+
[2025-07-21 11:21:00,529][mdl.fnn][INFO] - /bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 model with b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10/f0.e0.pt
11+
[2025-07-21 11:21:00,530][mdl.fnn][INFO] - Validation loss decreased (inf --> 117498.136230)
12+
[2025-07-21 11:21:33,113][mdl.fnn][INFO] - Fold 0/4, Epoch 1, Train Loss: 187871.2000
13+
[2025-07-21 11:21:33,114][mdl.fnn][INFO] - Fold 0/4, Epoch 1, Valid Loss: 117371.7754
14+
[2025-07-21 11:21:33,157][mdl.fnn][INFO] - /bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 model with b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10/f0.e1.pt
15+
[2025-07-21 11:21:33,157][mdl.fnn][INFO] - Validation loss decreased (117498.136230 --> 117371.775391)
16+
[2025-07-21 11:22:07,351][mdl.fnn][INFO] - Fold 0/4, Epoch 2, Train Loss: 187638.4172
17+
[2025-07-21 11:22:07,352][mdl.fnn][INFO] - Fold 0/4, Epoch 2, Valid Loss: 117269.4438
18+
[2025-07-21 11:22:07,405][mdl.fnn][INFO] - /bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 model with b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10/f0.e2.pt
19+
[2025-07-21 11:22:07,405][mdl.fnn][INFO] - Validation loss decreased (117371.775391 --> 117269.443848)
20+
[2025-07-21 11:22:39,949][mdl.fnn][INFO] - Fold 0/4, Epoch 3, Train Loss: 187403.3891
21+
[2025-07-21 11:22:39,949][mdl.fnn][INFO] - Fold 0/4, Epoch 3, Valid Loss: 117160.7539
22+
[2025-07-21 11:22:40,003][mdl.fnn][INFO] - /bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 model with b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10/f0.e3.pt
23+
[2025-07-21 11:22:40,003][mdl.fnn][INFO] - Validation loss decreased (117269.443848 --> 117160.753906)
24+
[2025-07-21 11:23:12,542][mdl.fnn][INFO] - Fold 0/4, Epoch 4, Train Loss: 187117.2883
25+
[2025-07-21 11:23:12,543][mdl.fnn][INFO] - Fold 0/4, Epoch 4, Valid Loss: 116995.8057
26+
[2025-07-21 11:23:12,595][mdl.fnn][INFO] - /bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 model with b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10/f0.e4.pt
27+
[2025-07-21 11:23:12,596][mdl.fnn][INFO] - Validation loss decreased (117160.753906 --> 116995.805664)
28+
[2025-07-21 11:23:46,929][mdl.fnn][INFO] - Fold 0/4, Epoch 5, Train Loss: 186815.2039
29+
[2025-07-21 11:23:46,930][mdl.fnn][INFO] - Fold 0/4, Epoch 5, Valid Loss: 116833.2056
30+
[2025-07-21 11:23:47,008][mdl.fnn][INFO] - /bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 model with b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10/f0.e5.pt
31+
[2025-07-21 11:23:47,008][mdl.fnn][INFO] - Validation loss decreased (116995.805664 --> 116833.205566)
32+
[2025-07-21 11:24:19,941][mdl.fnn][INFO] - Fold 0/4, Epoch 6, Train Loss: 186479.9781
33+
[2025-07-21 11:24:19,941][mdl.fnn][INFO] - Fold 0/4, Epoch 6, Valid Loss: 116651.4023
34+
[2025-07-21 11:24:19,983][mdl.fnn][INFO] - /bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 model with b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10/f0.e6.pt
35+
[2025-07-21 11:24:19,983][mdl.fnn][INFO] - Validation loss decreased (116833.205566 --> 116651.402344)
36+
[2025-07-21 11:24:54,059][mdl.fnn][INFO] - Fold 0/4, Epoch 7, Train Loss: 186089.3039
37+
[2025-07-21 11:24:54,060][mdl.fnn][INFO] - Fold 0/4, Epoch 7, Valid Loss: 116426.6377
38+
[2025-07-21 11:24:54,103][mdl.fnn][INFO] - /bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 model with b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10 saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/bnn.b30000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b.nmc10/f0.e7.pt
39+
[2025-07-21 11:24:54,103][mdl.fnn][INFO] - Validation loss decreased (116651.402344 --> 116426.637695)

‎docs/e2e_journal_experiment/cpu_gpu_usage/f0.test.e10.pred.eval.mean.csv‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
,mean
2+
P_2,0.733886544101867
3+
P_5,0.715666356677863
4+
P_10,0.7067386794477492
5+
recall_2,0.0004931126899019305
6+
recall_5,0.0011438657031803484
7+
recall_10,0.0021546982719466844
8+
ndcg_cut_2,0.7412278778051644
9+
ndcg_cut_5,0.7247470118446774
10+
ndcg_cut_10,0.7155556783560022
11+
map_cut_2,0.00045304813143719886
12+
map_cut_5,0.000910333803882852
13+
map_cut_10,0.0015637703569034206
14+
success_2,0.8075684956005437
15+
success_5,0.8584305028972029
16+
success_10,0.8873667644323628
17+
aucroc,0.6592047526698008
18+
skill_coverage_2,0.9894663423707051
19+
skill_coverage_5,0.9943486658559303
20+
skill_coverage_10,0.9988613873190748

‎docs/e2e_journal_experiment/cpu_gpu_usage/f0.test.pred.eval.mean.csv‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
,mean
2+
P_2,0.7761105944631232
3+
P_5,0.7600329065026095
4+
P_10,0.7393697689391234
5+
recall_2,0.000591188094492492
6+
recall_5,0.0013172216678159093
7+
recall_10,0.0023515005407380436
8+
ndcg_cut_2,0.7786319014626571
9+
ndcg_cut_5,0.7656160753168367
10+
ndcg_cut_10,0.7496899509048414
11+
map_cut_2,0.000548060192534427
12+
map_cut_5,0.0011034900974589972
13+
map_cut_10,0.001832411725398885
14+
success_2,0.830138064239216
15+
success_5,0.8680520781171758
16+
success_10,0.8929823306388154
17+
aucroc,0.7732016842776464
18+
skill_coverage_2,0.9932219758208776
19+
skill_coverage_5,0.9970849130839142
20+
skill_coverage_10,0.998503707942868

‎docs/e2e_journal_experiment/cpu_gpu_usage/file_not_find_error.png‎

260 KB
Loading

‎docs/e2e_journal_experiment/cpu_gpu_usage/fnn.bnn_merge_teams_w_same_skills=False.log‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
[2025-07-21 11:30:16,735][cmn.team][INFO] - Loading teamsvecs matrices from /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/teamsvecs.pkl ...
2+
[2025-07-21 11:30:17,221][cmn.team][INFO] - Loading indexes pickle from /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/indexes.pkl ...
3+
[2025-07-21 11:30:17,282][cmn.team][INFO] - Indexes pickle is loaded.
4+
[2025-07-21 11:30:17,282][cmn.team][INFO] - Teamsvecs matrices and indexes for skills (186385, 27), members (186385, 44774), and locations None are loaded.
5+
[2025-07-21 11:30:17,284][__main__][INFO] - Loading splits from /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/splits.pkl ...
6+
[2025-07-21 11:30:17,286][cmn.team][INFO] - Loading member-skill co-occurrence matrix (44774, 27) from /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/skillcoverage.pkl ...
7+
[2025-07-21 11:30:23,347][__main__][INFO] - Training team recommender instance mdl.fnn.Fnn ... 
8+
[2025-07-21 11:31:08,712][mdl.fnn][INFO] - Fold 0/4, Epoch 0, Train Loss: 7302.3172
9+
[2025-07-21 11:31:08,713][mdl.fnn][INFO] - Fold 0/4, Epoch 0, Valid Loss: 7083.7872
10+
[2025-07-21 11:31:08,743][mdl.fnn][INFO] - /fnn.b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b model with b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/fnn.b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b/f0.e0.pt
11+
[2025-07-21 11:31:08,743][mdl.fnn][INFO] - Validation loss decreased (inf --> 7083.787247)
12+
[2025-07-21 11:31:52,925][mdl.fnn][INFO] - Fold 0/4, Epoch 1, Train Loss: 6965.6673
13+
[2025-07-21 11:31:52,926][mdl.fnn][INFO] - Fold 0/4, Epoch 1, Valid Loss: 6867.9771
14+
[2025-07-21 11:31:52,957][mdl.fnn][INFO] - /fnn.b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b model with b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/fnn.b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b/f0.e1.pt
15+
[2025-07-21 11:31:52,957][mdl.fnn][INFO] - Validation loss decreased (7083.787247 --> 6867.977097)
16+
[2025-07-21 11:32:34,934][mdl.fnn][INFO] - Fold 0/4, Epoch 2, Train Loss: 6760.0003
17+
[2025-07-21 11:32:34,934][mdl.fnn][INFO] - Fold 0/4, Epoch 2, Valid Loss: 6763.6588
18+
[2025-07-21 11:32:34,964][mdl.fnn][INFO] - /fnn.b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b model with b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b saved at /home/msaeedi/OpeNTF/output/imdb/title.basics.tsv.mt10.ts2/fnn.b1000.e100.ns5.lr0.001.es5.h[128].speTrue.lbce.tpw1.tnw0.nsdunigram_b/f0.e2.pt
19+
[2025-07-21 11:32:34,964][mdl.fnn][INFO] - Validation loss decreased (6867.977097 --> 6763.658798)

‎docs/e2e_journal_experiment/cpu_gpu_usage/initial_eval_fnn.png‎

44.8 KB
Loading

‎src/evl/metric.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ def calculate_metrics(Y, Y_, per_instance=False, metrics=['P_2,5', 'recall_2,5',
1313
topk_idxes = np.argpartition(-Y_, kth=k - 1, axis=1)[:, :k]
1414
for i in range(Y.shape[0]):
1515
qrel['q' + str(i)] = {'d' + str(idx): 1 for idx in Y[i].nonzero()[1]}
16-
run['q' + str(i)] = {'d' + str(idx): k-rank for rank, idx in enumerate(topk_idxes[i])}
16+
run['q' + str(i)] = {'d' + str(idx): float(Y_[i][topk_idxes[i][j]]) for j, idx in enumerate(topk_idxes[i])}
1717
pbar.update(1)
1818
df = pd.DataFrame.from_dict(pytrec_eval.RelevanceEvaluator(qrel, set(metrics)).evaluate(run))
1919
df_mean = df.mean(axis=1).to_frame('mean')

0 commit comments

Comments
 (0)