News Aggregator

Creating a Command Line Tool With JBang and PicoCLI To Generate Release Notes

Aggregated on: 2024-08-06 13:07:52

Lately, I have been playing with JBang and PicoCLI, and I am pretty amazed at what we can do with these tools. I needed to create a script that would go to a specified repository on GitHub, check the commit range, and verify if any tickets were associated with them. Additionally, I wanted to check if the ticket was accepted and if the commit was approved or not. The idea was to integrate this script along with the CI/CD pipeline. While the traditional approach might involve using bash scripts or Python, as a Java developer, I feel more at home doing this in Java. This is where JBang comes into the picture. And since I want this to be a command-line tool, PicoCLI comes in handy.

Buh-Bye, Webpack and Node.js; Hello, Rails and Import Maps

Aggregated on: 2024-08-05 23:07:52

I enjoy spending time learning new technologies. However, often the biggest drawback of working with new technologies is the inevitable pain points that come with early adoption. I saw this quite a bit when I was getting up to speed with Web3 in “Moving From Full-Stack Developer To Web3 Pioneer.” As software engineers, we’re accustomed to accepting these early-adopter challenges when giving new tech a test drive. What works best for me is to keep a running list of notes and commands I’ve executed, since seemingly illogical steps don’t remain in my memory.

Go Serverless: Unleash Next-Gen Computing

Aggregated on: 2024-08-05 22:07:52

In the digital revolution, where bytes fly faster than thoughts, one concept is bringing a paradigm shift in the tech cosmos: serverless computing. The thought of dealing with servers often makes us freak out. Server maintenance, scalability issues, and huge infrastructure costs can all be part of our nightmares. This is where serverless computing can be a game-changer. It aims to virtually save modern-day technology trouble so we can just focus on coding. “Serverless” doesn't literally mean the servers completely vanish. Instead, they are hidden behind the curtains until summoned. Think of it like a magic genie that is always at your beck and call to grant your computing wishes without the hassles of hardware management.

Scaling Prometheus With Thanos

Aggregated on: 2024-08-05 21:07:52

Observability is a crucial pillar of any application, and monitoring is an essential component of it. Having a well-suited, robust monitoring system is crucial. It can help you detect issues in your application and provide insights once it is deployed. It aids in performance, resource management, and observability. Most importantly, it can help you save costs by identifying issues in your infrastructure. One of the most popular tools in monitoring is Prometheus. It sets a de facto standard with its straightforward and powerful query language PromQL, but it has limitations that make it unsuitable for long-term monitoring. Querying historical metrics in Prometheus is challenging because it is not designed for this purpose. Obtaining a global metrics view in Prometheus can be complex. While Prometheus can scale horizontally with ease on a small scale, it faces challenges when dealing with hundreds of clusters. In such scenarios, Prometheus requires significant disk space to store metrics, typically retaining data for around 15 days. For instance, generating 1TB of metrics per week can lead to increased costs when scaling horizontally, especially with the Horizontal Pod Autoscaler (HPA). Additionally, querying data beyond 15 days without downsampling further escalates these costs.

Reimagining AI: Ensuring Trust, Security, and Ethical Use

Aggregated on: 2024-08-05 20:07:52

The birth of AI dates back to the 1950s when Alan Turing asked, "Can machines think?" Since then, 73 years have passed, and technological advancements have led to the development of unfathomably intelligent systems that can recreate everything from images and voices to emotions (deep fake). These innovations have greatly benefited professionals in countless fields, be they data engineers, healthcare professionals, or finance personnel. However, this increased convergence of AI within our daily operations has also posed certain challenges and risks, and the assurance of reliable AI systems has become a growing concern nowadays.

Building an IoT-based Waste Management System: A Software Architect's Guide

Aggregated on: 2024-08-05 19:07:51

The Internet of Things is a network of physical devices. These devices can be anything, like smart bins or home appliances. They have sensors that collect information. They also have software that processes this information. These devices are connected to the internet. This allows them to share the data they collect. For example, a smart bin can tell how full it is and send this information to a cloud platform. We can use IoT to manage waste better. Sensors can gather data about waste levels. This helps in organizing waste collection more efficiently.

How To Setup OAuth JWT in the Salesforce Connector

Aggregated on: 2024-08-05 18:07:51

In this post, we'll explain all the steps required to connect a Mule application to Salesforce using the Salesforce connector with the OAuth JWT flow. You can also create your own certificate for the OAuth JWT flow with Salesforce or with OpenSSL (signed by a CA or self-signed). Both options are very well explained in the video at the conclusion of the article from Stefano Bernardini, MuleSoft Ambassador. In this post, we’ll be using a self-signed certificate created by Salesforce but, keep in mind, that for production environments, a certificate issued by a Trusted Certificate Authority is always recommended.

Streaming Data Joins: A Deep Dive Into Real-Time Data Enrichment

Aggregated on: 2024-08-05 17:07:51

Introduction to Data Joins In the world of data, a "join" is like merging information from different sources into a unified result. To do this, it needs a condition – typically a shared column – to link the sources together. Think of it as finding common ground between different datasets. In SQL, these sources are referred to as "tables," and the result of using a JOIN clause is a new table. Fundamentally, traditional (batch) SQL joins operate on static datasets, where you have prior knowledge of the number of rows and the content within the source tables before executing the Join. These join operations are typically simple to implement and computationally efficient. However, the dynamic and unbounded nature of streaming data presents unique challenges for performing joins in near-real-time scenarios.

Building a To-Do List With MongoDB and Golang

Aggregated on: 2024-08-05 16:07:51

Hi, there! Many have wondered how a simple task sheet or applications that provide such functionality work. In this article, I invite you to consider how you can write your small service in Go in a couple of hours and put everything in a database. Let's start our journey with Golang and MongoDB.

Free Tier API With Apache APISIX

Aggregated on: 2024-08-05 15:52:52

Lots of service providers offer a free tier of their service. The idea is to let you kick their service's tires freely. If you need to go above the free tier at any point, you'll likely stay on the service and pay. In this day and age, most services are online and accessible via an API. Today, we will implement a free tier with Apache APISIX. A Naive Approach I implemented a free tier in my post, "Evolving Your RESTful APIs: A Step-by-Step Approach," albeit in a very naive way. I copy-pasted the limit-count plugin and added my required logic.

Finding Your Voice: Navigating Tech as a Solo Female Engineer on Your Team

Aggregated on: 2024-08-05 15:07:51

For most of my career, I have been the only female engineer on my team. You may wonder, what’s so significant about that? As I navigated the tech industry as the only female engineer on my team, I often felt isolated and lonely. The lack of community and a sense of belonging led to a growing imposter syndrome, and unfortunately, many women in tech resonate with this feeling. Throughout my 5+ years of experience as a software engineer, I have realized the importance of having a strategy and a supportive network to navigate this landscape. Here are some of my tips to tackle this head-on:

Harnessing the Power of AWS Aurora for Scalable and Reliable Databases

Aggregated on: 2024-08-05 14:52:52

In the era of digital transformation, businesses require database solutions that provide scalability and reliability. AWS Aurora, a relational database that supports MySQL and PostgreSQL, has become a popular choice for companies looking for high performance, durability, and cost efficiency. This article delves into the benefits of AWS Aurora and presents a real-life example of how it is used in an online social media platform. Comparison of AWS Aurora: Benefits vs. Challenges Key Benefits Description Challenges Description High Performance and Scalability Aurora's design segregates storage and computing functions, delivering a bandwidth that is five times greater than MySQL and twice that of PostgreSQL. It guarantees consistent performance even during peak traffic periods by utilizing auto-scaling capabilities.

Overview of Classical Time Series Analysis: Techniques, Applications, and Models

Aggregated on: 2024-08-05 14:07:51

Time series data represents a sequence of data points collected over time. Unlike other data types, time series data has a temporal aspect, where the order and timing of the data points matter. This makes time series analysis unique and requires specialized techniques and models to understand and predict future patterns or trends. Applications of Time Series Modeling Time series modeling has a wide range of applications across various fields including:

Upgrading Spark Pipelines Code: A Comprehensive Guide

Aggregated on: 2024-08-05 13:52:51

In today's data-driven world, keeping your data processing pipelines up-to-date is crucial for maintaining efficiency and leveraging new features. Upgrading Spark versions can be a daunting task, but with the right tools and strategies, it can be streamlined and automated. Upgrading Spark pipelines is essential for leveraging the latest features and improvements. This upgrade process not only ensures compatibility with newer versions but also aligns with the principles of modern data architectures like the Open Data Lakehouse (Apache Iceberg). In this guide, we will discuss the strategic importance of Spark code upgrades and introduce a powerful toolkit designed to streamline this process.

Automation Resilience: The Hidden Lesson of the CrowdStrike Debacle

Aggregated on: 2024-08-05 13:07:51

The recent CrowdStrike debacle was a wake-up call of epic proportions. A simple null pointer error in a routine software update brought airlines, media companies, first responder networks, and many other enterprises to their knees. There is plenty of blame to spread around. The hapless developer who coded the bug, of course. But also the quality assurance team at CrowdStrike, CrowdStrike itself and its CEO, and Microsoft, whose systems were only too happy to roll over and blue screen.

Optimizing Software Quality: Unit Testing and Automation

Aggregated on: 2024-08-04 13:07:51

Unit testing is the first line of defense against bugs. This level of protection is essential as it lays the foundation for the following testing processes: integration tests, acceptance testing, and finally manual testing, including exploratory testing. In this article, I will shed some light on what differentiates unit testing from other methods and will bring examples of when we can or cannot do without unit testing. We'll also touch upon automation testing, which plays an important role in ensuring code reliability and quality.

Compound Interest in Software

Aggregated on: 2024-08-03 13:22:50

Many articles and posts online show how regular, small, incremental changes result in much larger gains over time. This article applies compound interest calculations to show that a 1% daily increase provides a 37.8-times improvement/growth/whatever over the year — the power of compound interest! Frustrating at first, but the gains grew each day — in the end very pleasing. Software engineers make decisions daily about the design and implementation of the solution being worked on: functionality, code structure, data validation, flow, error handling, security, performance, reliability, external dependencies, configurability, and more. Expected functionality and time-to-market often prioritize what is included and what is deferred/ignored: certain failures/errors are accepted, edge cases require manual implementation, etc. It's feature/implementation/tech debt that likely needs future remediation.

Industrial IoT: Exploring Microcontrollers for Robust Applications

Aggregated on: 2024-08-02 23:07:50

When we think of the Internet of Things (IoT), our imaginations are often limited to consumer applications: smart homes, smart transportation, smart offices, etc. However, these account for just the tip of the iceberg when it comes to IoT. Just beneath the surface lies a sprawling world of industrial IoT and industrial microcontrollers (MCUs). But just like an industrial solvent requires a formula special and distinct from your ordinary kitchen hand soap, the technical components of industrial IoT devices are starkly different from those that make your smart home HVAC system run smoothly. Nowhere is this distinction more sharp than when it comes to MCUs.

Leveraging Open-Source Contributions To Boost Your Freelancing Profile

Aggregated on: 2024-08-02 20:52:50

Building your reputation as a freelance developer means diligently improving your skills and reputation through real-world efforts. One of the best ways to do that is to contribute to open-source projects. How can you maximize that strategy for the best results? Align Projects With Skills and Professional Goals Developer communities and the internet at large are full of open-source projects that need developers’ expertise. Narrow the abundant possibilities by focusing on those that match your skills and career goals. Then, you will stay more motivated because the work gets you closer to long-term aspirations while increasing your marketability.

Criticality in Data Stream Processing and a Few Effective Approaches

Aggregated on: 2024-08-02 18:22:50

In the current fast-paced digital age, many data sources generate an unending flow of information: a never-ending torrent of facts and figures that, while perplexing when examined separately, provide profound insights when examined together. Stream processing can be useful in this situation. It fills the void between real-time data collecting and actionable insights. It's a data processing practice that handles continuous data streams from an array of sources. About Stream Processing Opposite to traditional batch data processing techniques, here, processing works on the data as it is produced in real time. In simple words, we can say processing data to get actionable insights when it is in motion before stationary on the repository. Data streaming processing is a continuous method of ingestion, processing, and, eventually, analyzing the data as it is generated from various sources.

Software Testing Errors To Look Out For (With Examples)

Aggregated on: 2024-08-02 16:22:50

Software will never be bug-free. However, it’s important to minimize the number of bugs such that the impact on the functionality and user experience of an application is minimized. Bugs could come up due to different reasons. In this article, we will discuss them from the perspective of software errors. These are the errors that also need attention during the testing phase. We can divide the errors during software development into two sections for easy understanding:

The C̶a̶k̶e̶ User Location Is a Lie!!!

Aggregated on: 2024-08-02 13:22:49

I recently sat in on a discussion about programming based on user location. Folks that are way smarter than me covered technical limitations, legal concerns, and privacy rights. It was nuanced, to say the least. So, I thought I’d share some details.

The One-Pixel Threat: How Minuscule Changes Can Fool Deep Learning Systems

Aggregated on: 2024-08-01 23:07:49

AI vulnerabilities: From medical diagnostics to autonomous vehicles, discover how changing a single pixel can compromise advanced deep learning models and explore the critical challenges to securing our AI-powered future. Introduction Deep learning (DL) is a fundamental component of Artificial Intelligence (AI). It aims to enable machines to perform tasks that require decision-making mechanisms that tend to approximate those of human reasoning. DL models are at the heart of many advanced applications, such as medical diagnostics and autonomous vehicle driving.

My Journey to Master SQL Data Analysis

Aggregated on: 2024-08-01 21:07:49

Dealing with sales data can feel like a task especially when you're trying to extract insights for important business decisions. In this article, we'll delve into the experience of a data analyst tackling these challenges and delve into why mastering SQL is key to achieving success. By the end, you'll grasp the role that data analysts play and how they can transform data into actionable information. Understanding the Role of a Data Analyst Data analysts often go unnoticed in organizations. They play a role. Their job involves sorting through volumes of data to identify patterns, trends, and insights that drive decision-making processes. Whether it's analyzing sales figures, customer feedback, or market research findings, data analysts are responsible for making sense of this information. However, this isn't a task. This demands a specific skill set that includes a solid grasp of SQL (Structured Query Language).

3 Reasons Why VPs of Engineering Are Choosing Low-Code

Aggregated on: 2024-08-01 19:22:49

I often hear the phrases "Low code is not flexible" and "Scaling is the major issue with low code" when talking to developers who have used it to build complex use cases like food ordering apps, e-commerce apps, and tools like that. For any type of application building, it's important to understand who you are developing it for, what is the ROI, and what they are going to achieve. Low code is something that can be best used when you want to build apps for internal use, rich data dashboards, CMS apps, or any app for operational needs within your organization. Developers, the primary creators, benefit hugely from pre-built UI components, ready databases, and API integrations. Important stat: 70% of new applications created by enterprises are expected to utilize low-code or no-code technologies, stats Gartner findings.

Navigating BNPL Integration: Key Steps and Best Practices for Developers

Aggregated on: 2024-08-01 17:22:49

Buy Now, Pay Later (BNPL) solutions are altering the way in which clients connect to payment systems by offering financial flexibility and enhancing the checkout experience. This guide offers developers and architects comprehensive knowledge on integrating your checkout process with BNPL as well as technical details, best practices, and common challenges. Importance of BNPL for Developers Enhanced Conversion Rates The abandonment of carts is reduced by the use of BNPL solutions that provide flexible payment options; hence, making it possible for users to complete their transactions. Users will not easily abandon their carts due to high upfront costs whenever payments can be broken down into small manageable amounts.

Agile-Regulated Industries?

Aggregated on: 2024-08-01 15:22:49

Are there agile-regulated industries, or is that an oxymoron? Recently, I received a project alert from an organization looking for a Scrum Master with “at least two years knowledge of information systems for rail and public transport travelers.”

How To Fix the OWASP Top 10 Vulnerability in Angular 18.1.1v

Aggregated on: 2024-08-01 13:22:49

The latest release of Angular, which is presently version 18.1.1, offers a wide range of features for developing robust and scalable web applications in Angular. However, safety continues to be of concern. In this article, we will discuss the configurations and see examples of how to mitigate these vulnerabilities in an Angular web application 18.1.1v in detail. 1. Injection Injection flaws (SQL, NoSQL, and OS command injection) occur when untrusted data is send to an interpreter situated among a machine or query.

Leveraging Time Series Databases for Cutting-Edge Analytics: Specialized Software for Providing Timely Insights at Scale

Aggregated on: 2024-08-01 12:22:49

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Database Systems: Modernization for Data-Driven Architectures. Time series data has become an essential part of data collection in various fields due to its ability to capture trends, patterns, and anomalies. Through continuous or periodic observation, organizations are able to track how key metrics are changing over time. This simple abstraction powers a broad range of use cases.

Merge Multiple PDFs in MuleSoft

Aggregated on: 2024-07-31 22:08:35

Suppose there is a scenario where you have to merge multiple PDFs into one PDF and send the merged PDF as a response back to the source system or store the merged PDF in a file location. In this article, you will learn how to do this. There is no such connector in MuleSoft that you can use to merge the PDFs. You will have to use the Java library org.apache.pdfbox.multipdf to merge the PDFs. It contains all necessary Java libraries for merging multiple PDFs into one. Implementation Step 1 Add the dependency as shown below in pom.xml:

Ensuring Data Integrity Through Anomaly Detection: Essential Tools for Data Engineers

Aggregated on: 2024-07-31 21:08:35

In the trending landscape of Machine Learning and AI, companies are tirelessly innovating to deliver cutting-edge solutions for their customers. However, amidst this rapid evolution, ensuring a robust data universe characterized by high quality and integrity is indispensable. While much emphasis is often placed on refining AI models, the significance of pristine datasets can sometimes be overshadowed. This article sets out to explore some of the essential tools required by organizations in the domain of data engineering to efficiently improve data quality and triage/analyze data for effective business-centric machine learning analytics, reporting, and anomaly detection. To illustrate these tools/frameworks and their importance, let us consider a scenario within the fintech industry.

Interference in A/B Tests

Aggregated on: 2024-07-31 20:08:35

A/B testing is the gold standard for online experimentation used by most companies to test out their product features. Whereas A/B test experimentation works just fine in most settings, it is particularly susceptible to interference bias, particularly in the case of online marketplaces or social networks. In this article, we aim to look at the situations with interference bias and some potential ways to mitigate its effect on evaluation. SUTVA, the Fundamental Assumption of A/B Testing and Its Violations One of the fundamental assumptions of A/B Testing is SUTVA — Stable Unit Treatment Value Assumption. The potential outcome of treatment in the randomization unit depends only on the treatment they receive and not on the treatments assigned to other subjects.

SingleStore Kai Support for MongoDB $vectorSearch

Aggregated on: 2024-07-31 19:08:38

SingleStore is a high-performance distributed SQL database and through SingleStore Kai extends its compatibility to MongoDB's vector search capabilities. By supporting MongoDB's $vectorSearch operator, SingleStore Kai provides efficient querying and indexing of vector data, such as embeddings used in ML and AI applications. Users can benefit from SingleStore's optimized data handling and Kai's vector search capabilities to perform similarity searches across large datasets. In this article, we'll see an example of how to use SingleStore Kai with vector data. The notebook file used in this article is available on GitHub.

Software Engineering Grief

Aggregated on: 2024-07-31 18:08:35

Human beings resist change. This is not personal opinion, it's our defense mechanism against uncertainty. If you narrow your target to software engineers and technologists - who, in almost all cases, are humans - the defense mechanism can become a survival mechanism.Successful companies rarely rest on their laurels and assume the money will flow in unimpeded by change. Instead, companies constantly analyze the current to define their vision for the future: How do economic and market conditions affect our ability to sell? Would new features or product offerings attract new customers while retaining existing customers? Could we more efficiently increase market share through acquisitions rather than internal product development? Are there partner opportunities beneficial to us? How can we expand the industries in which are products are sold?

Plot Your REST Endpoints Using grafana-infinity-datasource

Aggregated on: 2024-07-31 17:08:35

When it comes to observability Grafana is the go-to tool for visualization. A Grafana dashboard consists of various forms of visualizations which are usually backed by a database. This is not always the case. Sometimes instead of pushing the data from the database as is, you might want to refine the data. This cannot always be achieved through the functionalities the DB provides. For example, you might want to fetch results from a proprietary API. This is where the grafana-infinity-datasource plugin kicks in. With grafana-infinity-datasource, you can create visualizations based on JSON, XML, CSV, etc. You can issue an HTTP request to a REST API and plot the received data. Tutorial Let’s assume we have an eShop application. We will create a simple Python API using FastAPI to manage the items of the eShop and the purchase volume.

Two-Tower Model for Fraud Detection: A Comprehensive Guide

Aggregated on: 2024-07-31 16:53:35

Fraud detection is a very important task in most industries, especially in finance and e-commerce. The classic machine learning models struggle to handle the subtlety and complexity of patterns rooted in fraudulent behavior. The dual-tower or Siamese network model provides a powerful architecture to handle the complex tasks of parallel processing with two different sets of inputs, thus helping capture the intricate relationships between them. This article presents how to apply the two-tower model to fraud detection, with detailed explanations, code snippets, and practical illustrations. What Is a Two-Tower Model? It is an architecture that comprises two independent neural networks: one dealing with one type of input data and the second dealing with another. These two towers may work independently, although their output results are combined to make a single unified prediction. This architecture works very nicely in tasks involving finding relationships or similarities between two diverse sources of data.

Self-Compassion in Tech Teams: Building Strength and Teamwork

Aggregated on: 2024-07-31 16:08:35

Tech teams have a hard job. They deal with tight deadlines and tough problems. This can cause stress and self-criticism. But being kind to yourself can help. Self-compassion makes you feel better. It also helps the team work better together. This guide will explain what self-compassion is. It will also show why it’s good for tech teams and how to use it at work. Self-Compassion for Tech Teams Self-compassion is like being your own supportive friend. When things go wrong or you make mistakes, instead of being hard on yourself, treat yourself kindly. For tech teams, this means understanding hard tasks and handling mistakes gently. It helps you stay calm and focused. Dr. Kristin Neff talks about three parts of self-compassion:

Common CSS Breakpoints for Media Queries

Aggregated on: 2024-07-31 15:53:35

As technology is growing every day, the diversification in the types of electronic devices is also expanding. In the present time, a wide range of electronic devices exist, including tablets, laptops, desktops, smartphones, and even smartwatches. Tech giants are spending millions of dollars in their research and development to introduce new and improved electronic devices daily in the market. This has resulted in the daily continuous increase in the catalog of electronic devices.

Multimodal Search in AI Apps With PostgreSQL pgvector

Aggregated on: 2024-07-31 15:08:35

Large language models (LLMs) have significantly evolved beyond generating text responses to text prompts. These models are now trained to possess advanced capabilities like interpreting images and providing detailed descriptions from visual inputs. This gives users an even greater search capacity. In this article, I’ll demonstrate how to build an application with multimodal search functionality. Users of this application can upload an image or provide text input that allows them to search a database of Indian recipes. The application is built to work with multiple LLM providers, allowing users to choose between OpenAI, or a model running locally with Ollama. Text embeddings are then stored and queried in PostgreSQL using pgvector.

Vulnerability Management in DevOps Environments

Aggregated on: 2024-07-31 14:53:35

DevOps has become the groundwork for delivering top-notch applications quickly and efficiently in today’s agile development. Its efficiency and speed can also cause notable security threats if vulnerabilities are not managed properly. Sixty percent of data breaches succeed because organizations fail to apply known, available patches before the weaknesses are exploited.

Outlier Identification in Continuous Data Streams With Z-Score and Modified Z-Score in a Moving Window

Aggregated on: 2024-07-31 14:08:35

My name is Maksim Kupriianov, and for the past few years, I have been actively involved in network monitoring. This means sending network probes from different locations within an organization’s network, analyzing the responses with regard to packet loss percentage and response times, and identifying places in the network where something has gone wrong. The probes are sent and the results are analyzed continuously, with new data coming in every second. The analytics are conducted on a sliding window of several tens of seconds. During this process, local anomalies often arise. For instance, a server in one of the racks might start showing increased packet loss due to the server's high CPU utilization. On the one hand, this is already a problem that should be highlighted, but on the other hand, it is not caused by any network equipment failure and doesn’t affect network performance in general. A similar situation occurs with the response times: for various reasons, responses could be delayed even by seconds, whereas typically, the round-trip time of the requests does not exceed a few milliseconds.

Automatic 1111: Adding Custom APIs

Aggregated on: 2024-07-31 13:53:35

In our previous article, we thoroughly covered the fundamental aspects of an Automatic 1111. This included the setup process, enabling APIs, and other crucial considerations for optimal API usage. In this article, we will be discussing the Options API in depth and adding our own custom APIs in Automatic 1111 (1.9.3 version). Options API Options API is one of the most commonly used APIs for fetching and updating Automatic 1111 application configurations.

Differentiating Rate Limits in Apache APISIX

Aggregated on: 2024-07-31 13:08:35

In my talk "Evolving your APIs," I mention that an API Gateway is a reverse proxy "on steroids." One key difference between the former and the latter is that the API Gateway is not unfriendly to business logic. The poster child is rate-limiting. Rate-limiting is an age-old reverse proxy feature focused on protecting against DDoS attacks. It treats all clients the same and is purely technical. In this day and age, most API providers offer different subscription tiers; the higher the tier, the higher the rate limit, and the more you pay incidentally. It's not technical anymore and requires to differentiate between clients.

Just Use PostgreSQL, a Quick-Start Guide: Exploring Essential and Extended Capabilities of the Most Beloved Database

Aggregated on: 2024-07-31 12:53:35

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Database Systems: Modernization for Data-Driven Architectures. PostgreSQL has been in development and use for over 35 years. Throughout those years, the project that started as an open-source relational database for transactional workloads has turned into one of the most reliable and comprehensive database solutions for a variety of use cases. The depth and breadth of PostgreSQL's capabilities have become so rich that you might hear "just use Postgres" if you ask for advice about database options for your next application.

How To Generate Flame Graphs in Java

Aggregated on: 2024-07-30 23:08:34

Flame graphs became the de facto standard for investigating issues in today's applications (not only written in Java). The flame graphs can provide a lot of interesting insights and can give developers valuable hints to improve the execution of their applications. However, still, a lot of developers don't use them, even if generating flame graphs is easier than ever before. In this article, I will demonstrate a couple of examples of how easy it is in Java to get started with the investigation of your applications. If you don't know the details about flame graph visualization, please, visit a great article about flame graphs from Brendan Gregg.

GitOps: ArgoCD vs FluxCD

Aggregated on: 2024-07-30 22:08:34

Getting Started With GitOps Most organizations are always looking for ways to streamline their processes and improve efficiency by implementing automation. In recent years, software deployment has been done multiple times in a day rather than weeks or months. Organizations have moved from the waterfall models to hyper-agile methodologies and particularly due to the adoption of microservices architecture, teams are releasing their software much faster. To make this possible, GitOps implements a control-loop pattern more often seen in Kubernetes. GitOps offers a more consistent and reliable way to handle infrastructure and deployment. In this blog, we'll explore what GitOps is and why it's becoming increasingly popular among DevOps teams. We will also walk through popular GitOps tools like Argo CD and Flux CD.

How To Run Apache Spark on Kubernetes in Less Than 5 Minutes

Aggregated on: 2024-07-30 21:08:34

Tools like Ilum will go a long way in simplifying the process of installing Apache Spark on Kubernetes. This guide will take you step-by-step through how to run Spark well on your Kubernetes cluster. With Ilum, deploying, managing, and scaling Apache Spark clusters is easily and naturally done. Introduction Today, we will showcase how to get up and running with Apache Spark on K8s. There are many ways to do that, but most are complex and require several configurations. We will use Ilum since that will do all the cluster setup for us. In the next blog post, we will compare the usage with the Spark operator.

Implementing Multi-Level Caching in Java With NCache

Aggregated on: 2024-07-30 20:23:34

Multi-level caching is a technique used to improve the performance of applications by storing frequently accessed data in different cache layers. The cache layers generally lie at various levels of the application stack. It is a common practice to use a distributed cache like NCache to implement multi-level caching in applications. NCache provides a scalable and high-performance caching solution that can store data in memory across multiple servers. In addition to this, NCache provides a feature to enable local caching in client nodes to offer even faster data access.

Real-Time Anomaly Detection Using Large Language Models

Aggregated on: 2024-07-30 19:23:34

The capability to detect anomalies becomes important in the data-driven world of today and is a key component for various industries such as finance, healthcare, cybersecurity, and manufacturing. Anomalies can be a sign of fraud, system failings, security incidents, or other important events that require immediate attention. The volume, velocity, and variety of streaming data are difficult for traditional anomaly detection techniques to handle. On the other hand, recent developments in Large Language Models (LLMs) provide a new path to perform real-time anomaly detection. In this blog post, we discuss how LLMs can be used for anomaly detection on streaming data in detail with some examples. Anomaly Detection Anomalies are patterns in your data that do not conform to a well-defined notion of normal behavior.

Apache Iceberg Table Management

Aggregated on: 2024-07-30 18:23:37

1. What Are Iceberg and Table Layout for Iceberg? Apache Iceberg is a high-performance table format for large analytic datasets. It's designed to handle petabyte-scale data lakes with the reliability and efficiency needed for data analytics and big data workflows. Iceberg tables organize data into a consistent format that simplifies querying, updating, and managing data at scale. One of the main advantages of Iceberg table format is schema evolution, which allows updating the table schema without re-writing the data. However, all these advantages come at the cost of maintaining table metadata disjoint from data in metadata files which are updated for each table ops in a transaction while maintaining concurrency. A typical Iceberg table layout has: Manifest files: Store metadata about data files in the table, including their locations, sizes, and statistics. Snapshot files: Represent the state of the table at a given point in time. Each snapshot includes references to manifest files and data files. Data files: Contain the actual data in the table, typically stored in columnar formats like Parquet or ORC. Metadata files: Store global metadata about the table, such as schema, partitioning information, and properties. CRUD operations on table leads to generation of multiple snapshot files, manifest files, data files etc which can consume storage making the table operations inefficient.