Automated machine learning

Machine learning and data mining
Problems Classification Clustering Regression Anomaly detection AutoML Association rules Reinforcement learning Structured prediction Feature engineering Feature learning Online learning Semi-supervised learning Unsupervised learning Learning to rank Grammar induction
Supervised learning (classification • regression) Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
Clustering BIRCH CURE Hierarchical k-means Expectation–maximization (EM) DBSCAN OPTICS Mean-shift
Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA t-SNE
Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
Anomaly detection k-NN Local outlier factor
Artificial neural networks Autoencoder Deep learning DeepDream Multilayer perceptron RNN LSTM GRU) Restricted Boltzmann machine SOM Convolutional neural network U-Net
Reinforcement learning Q-learning SARSA Temporal difference (TD)
Theory Bias-variance dilemma Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
Machine-learning venues NIPS ICML ML JMLR ArXiv:cs.LG
Glossary of artificial intelligence Glossary of artificial intelligence
Related articles List of datasets for machine-learning research Outline of machine learning
Machine learning portal

Automated machine learning (AutoML) is the process of automating the end-to-end process of applying machine learning to real-world problems. In a typical machine learning application, practitioners must apply the appropriate data pre-processing, feature engineering, feature extraction, and feature selection methods that make the dataset amenable for machine learning. Following those preprocessing steps, practitioners must then perform algorithm selection and hyperparameter optimization to maximize the predictive performance of their final machine learning model. As many of these steps are often beyond the abilities of non-experts, AutoML was proposed as an artificial intelligence-based solution to the ever-growing challenge of applying machine learning. Automating the end-to-end process of applying machine learning offers the advantages of producing simpler solutions, faster creation of those solutions, and models that often outperform models that were designed by hand.

Targets of automation

Automated machine learning can target various stages of the machine learning process:

• Automated data preparation and ingestion (from raw data and miscellaneous formats)
  • Automated column type detection; e.g., boolean, discrete numerical, continuous numerical, or text
  • Automated column intent detection; e.g., target/label, stratification field, numerical feature, categorical text feature, or free text feature
  • Automated task detection; e.g., binary classification, regression, clustering, or ranking
• Automated feature engineering
  • Feature selection
  • Feature extraction
  • Meta learning and transfer learning
  • Detection and handling of skewed data and/or missing values
• Automated model selection
• Hyperparameter optimization of the learning algorithm and featurization
• Automated pipeline selection under time, memory, and complexity constraints
• Automated selection of evaluation metics / validation procedures
• Automated problem checking
  • Leakage detection
  • Misconfiguration detection
• Automated analysis of results obtained
• User interfaces and visualizations for automated machine learning

Examples

Notable platforms tackling various stages of AutoML:

Hyperparameter optimization and model selection

• Auto-WEKA is a Bayesian hyperparameter optimization layer on top of WEKA.
• auto-sklearn is a Bayesian hyperparameter optimization layer on top of scikit-learn.

Full pipeline optimization

• TPOT is a Python library that automatically creates and optimizes full machine learning pipelines using genetic programming.
• TransmogrifAI is a Scala/SparkML library created by Salesforce for automated data cleansing, feature engineering, model selection, and hyperparameter optimization
• RECIPE is a framework based on grammar-based genetic programming that builds customized scikit-learn classification pipelines.

Deep neural network architecture search

• Auto Keras is an open-source python package for neural architecture search.

See also

• Neural architecture search
• Hyperparameter optimization
• Model selection
• Neuroevolution
• Self-tuning