In this tutorial, we will go through into the process of web scraping, illustrating how to extract data from a website and then convert it into a CSV file using Python. We will start with a fundamental example that showcases the extraction and transformation of data from a website to…

Choosing between YAML and properties files in a Spring Boot application depends on your preferences and the complexity of your configuration needs. Here are some considerations to help you decide: YAML (YAML Ain’t Markup Language): Advantages: Readability: YAML is often considered more human-readable and intuitive compared to properties files. …

The Spark submit command is used to submit a Spark application to a cluster. It accepts a number of arguments, including four important properties: spark-submit \ --master <master-url> \ --deploy-mode <deploy-mode> \ --conf <key<=<value> \ # memory configs --driver-memory <value> \ --executor-memory…

Lets understand index by a simple example. Imagine you have a big collection of cars , and you want to find a specific car , let’s say a Red Ferrari from this collection. You could go through each car , one by one, until you find your Ferrari . …

First thing first JPA (Java persistence API) is not a tool or not a framework its just Enables seamless interaction between Java Objects and Relational Databases for data access, persistence, and management. Spring Boot Starter Data JPA is a convenience dependency that provides all of the necessary dependencies to use…

An API gateway is a software intermediary that sits between client applications and backend services. It acts as a single point of entry for all API requests It routes those requests to the appropriate backend service. …

Data pipelines can be designed in many ways, but there are a few common patterns that are often used. In this article, I will discuss three of the most popular data pipeline design patterns: ETL, ELT, and CDC. ETL (Extract, Transform, Load) ETL is a traditional data pipeline design pattern that involves extracting data…

Exception handling is a critical aspect of developing robust applications. In Spring, when working with filters, it’s essential to know how to effectively handle exceptions that may occur within the doFilter() method. In this article, we will explore best practices for managing exceptions in this context. Using the Global Exception Handler: Whenever an exception occurs…

Imagine a special system that’s really good at handling constantly changing information. This system, called an incremental data pipeline, helps process data as it’s created or changed. It’s like having a super-fast way to make sense of all the information that’s always evolving. Why It Matters ?