Fpl-data fetches data from the Fantasy Premier League servers, cleans and transforms the data, and then stores it in MySQL and Redis.
Data processing occurs in three scenarios: every day, match day, and every gameweek, implemented using Spring Boot schedules for these tasks.
In the initial stages, the services in this project were originally designed within the FPL project. As the project expanded, these services were extracted and relocated to this dedicated project, a strategic move aimed at enhancing maintenance and scalability.
At present, these two projects operate independently, with Fpl-data handling data, while FPL is responsible for delivering services to users. This architectural setup has proven effective for over two years.
- Java 20 + Spring Boot 3
- MySQL for data storage
- Redis for caching
- Maven for build and dependency management
- Logback for logging
- Jasypt for encryption and decryption of sensitive data
- Jenkins for CI/CD
flowchart LR
Fpl_Data -. fetch .-> FPL_Servers
Fpl_Data -- store --> id1 & id2
subgraph PERSIST
id1[(MySQL)]
id2[(Redis)]
style id1 fill:#f9f
style id2 fill:#0f0
end
Spring Boot schedules prove effective for this project, and I opted not to introduce additional scheduling frameworks such as xxl-job, Quartz, or others to maintain simplicity.
These tasks handle the data that needs to be fetched and processed every day.
Given that the Premier League may adjust the schedule at any time during the season, the event data needs daily updates to ensure alignment with the real schedule in the database.
One of the most important components of the FPL game is the player's value, which affects the strategy of playing the game massively. The player value will be updated at 9:30 am BST, so the purpose of this task is to fetch the changing of the player value in time and show to the user.
Simultaneously with the player value update, the player value info is also refreshed. This information includes the likelihood of playing in the next round, transfer in and out numbers, and more.
Player stats, such as goals, assists, and clean sheets, are updated daily.
As FPL allows players to change their team name and username, this task ensures the accurate representation of this information in the database, maintaining synchronisation with real data.
Tournaments in LetLetMe allow users to establish custom tournaments to compete with friends. This task updates tournament information, accommodating changes in basic details.
These tasks manage data changes during match days, primarily focusing on data dependent on match results
Among the critical tasks in the project, the Event Live Task provides real-time data for every player in the event. This data encompasses the player's points, minutes played, goals, assists, and more. It forms the basis for calculating live scores for each user's FPL team, contributing to live scores and rankings within their respective tournaments. Due to server limitations, the update period for this task is set to 5 minutes, deviating from the real-time updates during actual matches.
The Event Live Summary task consolidates player event live stats from gameweek 1 to the current gameweek into a single record.
The Event Live Explain Task offers detailed insights into a player's live event data, explaining how they accumulated points. This includes information on playing minutes, goals, assists, clean sheets, and more.
This task updates the overall results of the event, encompassing highest scores, chips played, most-captained players, most-transferred players in the event, and more.
These tasks are designed to update the tournaments results for every player in the tournaments after every match day, including:
- player's points, ranking details
- player's picks, chips played, starting XI, and captain
- player's transfers, hits, and additional details
Reports are generated for each tournament after every gameweek, showcasing the ranking, points, and other relevant details. These reports play a crucial role in data analysis and visualisation for users.
MySQL serves as the relational database for this project, and the MyBatis_Plus framework is selected to augment the utilisation of MyBatis within the project. For a detailed overview of the database structure, you can refer to the ER diagram provided in the FPL project Readme.
The usage of AOP in the project is to log the service behaviours without modifying the business logic.
To facilitate better maintenance, the logging in the project is designed to be flexible and user-friendly. The project utilises Logback as the logging framework and Slf4j as the logging facade. The logback configuration file, logback-spring.xml, is located in the resources folder and is tailored for flexibility and ease of use.
Logs are separated into three files:
- interface.log: Time-based, rolled daily, used for monitoring the requests and responses of the interfaces, specifically HTTP calls between this project and others.
- task.log: Time-based, rolled daily, used for monitoring scheduled tasks.
- fpl-data.log: Time-based, rolled daily, used for monitoring the business logic of the project.
This project is deployed on a Linux server. The deployment process is automated using Jenkins.
The deployment process is as follows:
- Jenkins pulls the code from the GitHub repository.
- Jenkins builds the project using Maven.
- Jenkins publishes the built artifact to the server via Plugin publish Over SSH.
Fpl-data provides the ability to fetch transformed and cleaned Fantasy Premier League data.
There are primarily two projects utilising this data:
- FPL: The backend project for the website and WeChat Mini Program LetLetMe. It uses the data processed by Fpl-data to offer services for users to view their scoring, ranking, and summary reports for their Fantasy Premier League team.
- TelegramBot: A Java-based project that offers users the service to retrieve their FPL data via the Telegram bot letletme.