The solution in GCP to gather data from different data sources (SQL, NoSQL, REST) related to the API platform. Also, data were transformed and prepared for reporting and an analytic tool like Tableau or Data Studio. The developed solution includes the reliable and flexible architecture of the data pipeline, the data model design and data dictionary for reporting, the framework for data extraction and transformation from different sources, the orchestration engine for the data pipeline, the collection of reports in Tableau.
We plan to talk about the solution architecture and implementation of the framework for data extraction on Java and Apache beam.
