Outlier detection, also known as anomaly detection, is a common task for many data science teams. It is the process of identifying data points that have extreme values compared to the rest of the distribution. Outlier detection has a wide range of applications including data quality monitoring, identifying price arbitrage in finance, detecting cybersecurity attacks, healthcare fraud detection, banknote counterfeit detection and more.

In each of these applications, outliers correspond to events that are rare or uncommon. For example, in the case of cybersecurity attacks, most of the events represented in the data will not reflect an actual attack. …

For data scientists, a key part of interpreting machine learning models is understanding which factors impact predictions. In order to effectively use machine learning in their decision-making processes, companies need to know which factors are most important. For example, if a company wants to predict the likelihood of customer churn, they might also want to know what exactly drives a customer to leave a company. In this example, the model might indicate that customers who purchase products that rarely go on sale are much more likely to stop purchasing. Armed with this knowledge, a company can make smarter pricing decisions…

Across industries, organizations commonly use time series data, which means any information collected over a regular interval of time, in their operations. Examples include daily stock prices, energy consumption rates, social media engagement metrics and retail demand, among others. Analyze time series data yields insights like trends, seasonal patterns and forecasts into future events that can help generate profits. For example, by understanding the seasonal trends in demand for retail products, companies can plan promotions to maximize sales throughout the year.

When analyzing time series data, you should undertake a number of steps. First, you need to check for stationarity…

Exploratory data analysis (EDA) is an important part of every data scientist’s workflow. EDA allows data scientists to summarize the most important characteristics of the data they’re working with. In the case of financial data analysis, this includes generating simple summary statistics such as standard deviation in returns and average returns, visualizing relationships between stocks through correlation heatmaps, generating stock price time series plots, boxplots, and more.

Let’s consider the analysis of three stocks: Amazon (AMZN), Google (GOOGL) and Apple (AAPL). We’ll see how to perform simple exploratory data analysis of these stocks by generating summary statistics and visualizations, risk…

Building stable, accurate and interpretable machine learning models is an important task for many companies across industries. Machine learning model predictions have to be stable in time as the underlying training data is updated. Drastic changes in data due to unforeseen events can lead to significant deterioration in model performance. Model hyperparameter tuning can help make necessary changes to machine learning models that account for statistical changes in data over time. It is also important to understand the various ways of testing your models depending on how much data you have and consequently the stability of your model predictions. Further…

Companies have great interest in clearly communicating their ML-based predictive analytics to their clients. No matter how accurate a model is, clients want to know how machine learning models make predictions from data. For example, if a subscription-based company is interested in finding customers who are at high risk of canceling their subscriptions, they can use their historical customer data to predict the likelihood of someone leaving.

From there, they would want to analyze the factors that drive this event. By understanding the driving factors, they can take actions like targeted promotions or discounts to prevent the customer from leaving…

Machine learning classification is a type of supervised learning in which an algorithm maps a set of inputs to discrete output. Classification models have a wide range of applications across disparate industries and are one of the mainstays of supervised learning. This is because, across industries, many analytical questions can be framed in terms of mapping inputs to a discrete set of outputs. The simplicity of defining a classification problem makes classification models versatile and industry agnostic.

An important part of building classification models is evaluating model performance. In short, data scientists need a reliable way to test approximately how…

Raw text data often comes in a form that is difficult to use directly for analysis and it often requires text processing methods. Text processing is the practice of automating the generation and manipulation of text. It can be used for many data manipulation tasks including feature engineering from text, data wrangling, web scraping, search engines and much more. In this tutorial, we’ll take a look at how to use the Pandas library to perform some important data wrangling tasks.

Data wrangling is the process of gathering and transforming data to address an analytical question. It’s also often the most…