2023-07-08T1206H UTC+8: Optimization Problem
The second week at work I have to ask for a task after setting up my developer workspace and the first problem that I was given was to optimize some base code for one of the projects.
Technically the regular senior and technical lead dictated on what has to be done. The first problem was to implement a pagination for a potential large result set coming from a DB query that could span a million records or more. In order to not clog the heap memory a pagination has to be implemented. Had to understand the whole base code that was about some scheduled task and do some simple calculation in order to come up with the number of loops to call for the pagination. Later on had to move the pagination stuffs on a dedicated pagination utility class.
The next problem to solve then was to implement a batch saving to the RDBMS and Elastic Search since the base code saves stuffs per records from the original one time database query. The batch saving has to be implemented within the scope of each pagination call.
The final problem to solve and is still in progress and is about thread pooling and although implemented I noticed some batch saves to the ES are being dropped more frequently than the batch saves to the RDBMS. Debugging the stuffs stepping each time on the code completes the batch saves to both the DB and the ES otherwise without stepping things gets dropped. I'm thinking it maybe due to the thread pool executor being shutdown after the thread task has been submitted.