The week's pick
Reseach Article
Enhancing Recommendations of Items by Making Some Changes in Layers of BERT Model
| International Journal of Computer Applications |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 186 - Number 3 |
| Year of Publication: 2024 |
| Authors: Ashima Malik, S. Srinivasan, Piyush Prakash |
10.5120/ijca2024923366
|
Ashima Malik, S. Srinivasan, Piyush Prakash . Enhancing Recommendations of Items by Making Some Changes in Layers of BERT Model. International Journal of Computer Applications. 186, 3 ( Jan 2024), 28-36. DOI=10.5120/ijca2024923366
Abstract
Introducing an innovative methodology for modeling user behavior sequences in recommendation systems, this paper proposes the use of a bidirectional self-attention network and Cloze task, drawing inspiration from Bidirectional Encoder Representations from Transformers (BERT) to enhance the recommendations of products on e-commerce websites. Traditional recommendation system models that are unidirectional have limitations, mainly in the power of hidden representations and rigid ordering of historical user interactions. Overcoming these limitations, the suggested BERT4Rec model is bidirectional, offering the context from both directions. The paper suggests utilizing the Cloze task to prevent data leakages from bidirectional conditioning. This includes masking random components within the input sequences and predicting them based on their nearby context. Comprehensive experiments are conducted, resulting in consistently better outcomes than state-of-the-art comparable options across four datasets. This exploration sets the groundwork by introducing the Cloze objective and deep bidirectional sequential modeling to the recommendation system field. Furthermore, the study is a foundation for future studies investigating explicit user modeling and incorporating item features.
References
- Akhtyamova, L., 2020, April. Named entity recognition in Spanish biomedical literature: Short review and BERT model. In 2020 26th Conference of Open Innovations Association (FRUCT) (pp. 1-7). IEEE. https://arrow.tudublin.ie/cgi/viewcontent.cgi?article=1016&context=ittscicon
- Balázs Hidasi and Alexandros Karatzoglou. 2018. Recurrent Neural Networks with Top-k Gains for Session-based Recommendations. In Proceedings of CIKM. ACM, New York, NY, USA, 843–852.
- Ciniselli, M., Cooper, N., Pascarella, L., Poshyvanyk, D., Di Penta, M. and Bavota, G., 2021, May. An empirical study on the usage of BERT models for code completion. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) (pp. 108-119). IEEE. https://arxiv.org/pdf/2103.07115
- F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (Dec. 2015), 19 pages.
- Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. In Deep Learning and Representation Learning Workshop.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of CVPR. 770–778.
- Lee, J.S. and Hsiang, J., 2019. Patentbert: Patent classification with fine-tuning a pre-trained bert model. arXiv preprint arXiv:1906.02124. https://arxiv.org/pdf/1906.02124
- Lin, J., Liu, Y., Zeng, Q., Jiang, M. and Cleland-Huang, J., 2021, May. Traceability transformed: Generating more accurate links with pre-trained bert models. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) (pp. 324-335). IEEE. https://arxiv.org/pdf/2102.04411
- Lu, W., Jiao, J. and Zhang, R., 2020, October. Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 2645-2652). https://arxiv.org/pdf/2002.06275
- Mozafari, M., Farahbakhsh, R. and Crespi, N., 2020. Hate speech detection and racial bias mitigation in social media based on BERT model. PloS one, 15(8), p.e0237861. https://doi.org/10.1371/journal.pone.0237861
- Nagy, A., Bial, B. and Ács, J., 2021. Automatic punctuation restoration with BERT models. arXiv preprint arXiv:2101.07343. https://arxiv.org/pdf/2101.07343
- Nozza, D., Bianchi, F. and Hovy, D., 2020. What the [mask]? making sense of language-specific BERT models. arXiv preprint arXiv:2003.02912. https://arxiv.org/pdf/2003.02912
- Petrov, A. and Macdonald, C., 2022, September. A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation. In Proceedings of the 16th ACM Conference on Recommender Systems (pp. 436-447). https://arxiv.org/pdf/2207.07483
- Qiao, Y., Zhu, X. and Gong, H., 2022. BERT-Kcr: prediction of lysine crotonylation sites by a transfer learning method with pre-trained BERT models. Bioinformatics, 38(3), pp.648-654. http://structpred.life.tsinghua.edu.cn/pdf/10.1093_bioinformatics_btab712.pdf
- Risch, J. and Krestel, R., 2020, May. Bagging BERT models for robust aggression identification. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying (pp. 55-61). https://aclanthology.org/2020.trac-1.9.pdf
- Rogers, A., Kovaleva, O. and Rumshisky, A., 2021. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8, pp.842-866. https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00349/1923281/tacl_a_00349.pdf
- Ruining He and Julian McAuley. 2016. Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation. In Proceedings of ICDM. 191–200.
- Ruining He, Wang-Cheng Kang, and Julian McAuley. 2017. Translation-based Recommendation. In Proceedings of RecSys. ACM, New York, NY, USA, 161–169.
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (Nov. 1997), 1735–1780.
- Shi, P. and Lin, J., 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255. https://arxiv.org/pdf/1904.05255
- Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W. and Jiang, P., 2019, November. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management (pp. 1441-1450). https://arxiv.org/pdf/1904.06690.pdf%EF%BC%89
- Sun, S., Cheng, Y., Gan, Z. and Liu, J., 2019. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355. https://arxiv.org/pdf/1908.09355
- Tsai, H., Riesa, J., Johnson, M., Arivazhagan, N., Li, X. and Archer, A., 2019. Small and practical BERT models for sequence labeling. arXiv preprint arXiv:1909.00100. https://arxiv.org/pdf/1909.00100
- Wang, Z., Ng, P., Ma, X., Nallapati, R. and Xiang, B., 2019. Multi-passage bert: A globally normalized bert model for open-domain question answering. arXiv preprint arXiv:1908.08167. https://arxiv.org/pdf/1908.08167.pdf)
- Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In Proceedings of WWW. 173–182.
Index Terms
Keywords