A language model is a computational model that predicts sequences in natural language. Language models are useful for a variety of tasks, including speech recognition, machine translation, natural language generation (generating more human-like text), optical character recognition, route optimization, handwriting recognition, grammar induction, information retrieval and disaster response. Large language models (LLMs), currently their most advanced form as of 2026, are predominantly based on transformers trained on larger datasets (frequently using texts scraped from the public internet). They have superseded recurrent neural network-based models, which had previously superseded the purely statistical models, such as the word n-gram language model. == History == Noam Chomsky did pioneering work on language models in the 1950s by developing a theory of formal grammars. In 1980, statistical approaches were explored and found to be more useful for many purposes than rule-based formal grammars. Discrete representations like word n-gram language models, with probabilities for discrete combinations of words, made significant advances. In the 2000s, continuous representations for words, such as word embeddings, began to replace discrete representations. Typically, the representation is a real-valued vector that encodes a word’s meaning such that words closer in vector space are similar in meaning and common relationships between words, such as plurality or gender, are preserved. == Pure statistical models == In 1980, the first significant statistical language model was proposed, and during the decade IBM performed 'Shannon-style' experiments, in which potential sources for language modeling improvement were identified by observing and analyzing the performance of human subjects in predicting or correcting text. === Models based on word n-grams === === Exponential === Maximum entropy language models encode the relationship between a word and the n-gram history using feature functions. The equation is P ( w m ∣ w 1 , … , w m − 1 ) = 1 Z ( w 1 , … , w m − 1 ) exp ( a T f ( w 1 , … , w m ) ) {\displaystyle P(w_{m}\mid w_{1},\ldots ,w_{m-1})={\frac {1}{Z(w_{1},\ldots ,w_{m-1})}}\exp(a^{T}f(w_{1},\ldots ,w_{m}))} where Z ( w 1 , … , w m − 1 ) {\displaystyle Z(w_{1},\ldots ,w_{m-1})} is the partition function, a {\displaystyle a} is the parameter vector, and f ( w 1 , … , w m ) {\displaystyle f(w_{1},\ldots ,w_{m})} is the feature function. In the simplest case, the feature function is just an indicator of the presence of a certain n-gram. It is helpful to use a prior on a {\displaystyle a} or some form of regularization. The log-bilinear model is another example of an exponential language model. === Skip-gram model === == Neural models == === Recurrent neural network === Continuous representations or embeddings of words are produced in recurrent neural network-based language models (known also as continuous space language models). Such continuous space embeddings help to alleviate the curse of dimensionality, which is the consequence of the number of possible sequences of words increasing exponentially with the size of the vocabulary, further causing a data sparsity problem. Neural networks avoid this problem by representing words as non-linear combinations of weights in a neural net. === Large language models === Although sometimes matching human performance, it is not clear whether they are plausible cognitive models. At least for recurrent neural networks, it has been shown that they sometimes learn patterns that humans do not, but fail to learn patterns that humans typically do. == Evaluation and benchmarks == Evaluation of the quality of language models is mostly done by comparison to human created sample benchmarks created from typical language-oriented tasks. Other, less established, quality tests examine the intrinsic character of a language model or compare two such models. Since language models are typically intended to be dynamic and to learn from data they see, some proposed models investigate the rate of learning, e.g., through inspection of learning curves. Various data sets have been developed for use in evaluating language processing systems. These include: Massive Multitask Language Understanding (MMLU) Corpus of Linguistic Acceptability GLUE benchmark Microsoft Research Paraphrase Corpus Multi-Genre Natural Language Inference Question Natural Language Inference Quora Question Pairs Recognizing Textual Entailment Semantic Textual Similarity Benchmark SQuAD question answering Test Stanford Sentiment Treebank Winograd NLI BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OpenBookQA, NaturalQuestions, TriviaQA, RACE, BIG-bench hard, GSM8k, RealToxicityPrompts, WinoGender, CrowS-Pairs
Situated approach (artificial intelligence)
In artificial intelligence research, the situated approach builds agents that are designed to behave effectively successfully in their environment. This requires designing AI "from the bottom-up" by focussing on the basic perceptual and motor skills required to survive. The situated approach gives a much lower priority to abstract reasoning or problem-solving skills. The approach was originally proposed as an alternative to traditional approaches (that is, approaches popular before 1985 or so). After several decades, classical AI technologies started to face intractable issues (e.g. combinatorial explosion) when confronted with real-world modeling problems. All approaches to address these issues focus on modeling intelligences situated in an environment. They have become known as the situated approach to AI. == Emergence of a concept == === From traditional AI to Nouvelle AI === During the late 1980s, the approach now known as Nouvelle AI (Nouvelle means new in French) was pioneered at the MIT Artificial Intelligence Laboratory by Rodney Brooks. As opposed to classical or traditional artificial intelligence, Nouvelle AI purposely avoided the traditional goal of modeling human-level performance, but rather tries to create systems with intelligence at the level of insects, closer to real-world robots. But eventually, at least at MIT new AI did lead to an attempt for humanoid AI in the Cog Project. === From Nouvelle AI to behavior-based and situated AI === The conceptual shift introduced by nouvelle AI flourished in the robotics area, given way to behavior-based robotics (BBR), a methodology for developing AI based on a modular decomposition of intelligence. It was made famous by Rodney Brooks: his subsumption architecture was one of the earliest attempts to describe a mechanism for developing BBAI. It is extremely popular in robotics and to a lesser extent to implement intelligent virtual agents because it allows the successful creation of real-time dynamic systems that can run in complex environments. For example, it underlies the intelligence of the Sony Aibo and many RoboCup robot teams. Realizing that in fact all these approaches were aiming at building not an abstract intelligence, but rather an intelligence situated in a given environment, they have come to be known as the situated approach. In fact, this approach stems out from early insights of Alan Turing, describing the need to build machines equipped with sense organs to learn directly from the real-world instead of focusing on abstract activities, such as playing chess. == Definitions == Classically, a software entity is defined as a simulated element, able to act on itself and on its environment, and which has an internal representation of itself and of the outside world. An entity can communicate with other entities, and its behavior is the consequence of its perceptions, its representations, and its interactions with the other entities. === AI loop === Simulating entities in a virtual environment requires simulating the entire process that goes from a perception of the environment, or more generally from a stimulus, to an action on the environment. This process is called the AI loop and technology used to simulate it can be subdivided in two categories. Sensorimotor or low-level AI deals with either the perception problem (what is perceived?) or the animation problem (how are actions executed?). Decisional or high-level AI deals with the action selection problem (what is the most appropriate action in response to a given perception, i.e. what is the most appropriate behavior?). === Traditional or symbolic AI === There are two main approaches in decisional AI. The vast majority of the technologies available on the market, such as planning algorithms, finite-state machines (FSA), or expert systems, are based on the traditional or symbolic AI approach. Its main characteristics are: It is top-down: it subdivides, in a recursive manner, a given problem into a series of sub-problems that are supposedly easier to solve. It is knowledge-based: it relies on a symbolic description of the world, such as a set of rules. However, the limits of traditional AI, which goal is to build systems that mimic human intelligence, are well-known: inevitably, a combinatorial explosion of the number of rules occurs due to the complexity of the environment. In fact, it is impossible to predict all the situations that will be encountered by an autonomous entity. === Situated or behavioral AI === In order to address these issues, another approach to decisional AI, also known as situated or behavioral AI, has been proposed. It does not attempt to model systems that produce deductive reasoning processes, but rather systems that behave realistically in their environment. The main characteristics of this approach are the following: It is bottom-up: it relies on elementary behaviors, which can be combined to implement more complex behaviors. It is behavior-based: it does not rely on a symbolic description of the environment, but rather on a model of the interactions of the entities with their environment. The goal of situated AI is to model entities that are autonomous in their environment. This is achieved thanks to both the intrinsic robustness of the control architecture, and its adaptation capabilities to unforeseen situations. === Situated agents === In artificial intelligence and cognitive science, the term situated refers to an agent which is embedded in an environment. The term situated is commonly used to refer to robots, but some researchers argue that software agents can also be situated if: they exist in a dynamic (rapidly changing) environment, which they can manipulate or change through their actions, and which they can sense or perceive. Examples might include web-based agents, which can alter data or trigger processes (such as purchases) over the Internet, or virtual-reality bots which inhabit and change virtual worlds, such as Second Life. Being situated is generally considered to be part of being embodied, but it is useful to consider each perspective individually. The situated perspective emphasizes that intelligent behavior derives from the environment and the agent's interactions with it. The nature of these interactions are defined by an agent's embodiment. == Implementation principles == === Modular decomposition === The most important attribute of a system driven by situated AI is that the intelligence is controlled by a set of independent semi-autonomous modules. In the original systems, each module was actually a separate device or was at least conceived of as running on its own processing thread. Generally, though, the modules are just abstractions. In this respect, situated AI may be seen as a software engineering approach to AI, perhaps akin to object oriented design. Situated AI is often associated with reactive planning, but the two are not synonymous. Brooks advocated an extreme version of cognitive minimalism which required initially that the behavior modules were finite-state machines and thus contained no conventional memory or learning. This is associated with reactive AI because reactive AI requires reacting to the current state of the world, not to an agent's memory or preconception of that world. However, learning is obviously key to realistic strong AI, so this constraint has been relaxed, though not entirely abandoned. === Action selection mechanism === The situated AI community has presented several solutions to modeling decision-making processes, also known as action selection mechanisms. The first attempt to solve this problem goes back to subsumption architectures, which were in fact more an implementation technique than an algorithm. However, this attempt paved the way to several others, in particular the free-flow hierarchies and activation networks. A comparison of the structure and performances of these two mechanisms demonstrated the advantage of using free-flow hierarchies in solving the action selection problem. However, motor schemas and process description languages are two other approaches that have been used with success for autonomous robots. == Notes and references == Arsenio, Artur M. (2004) Towards an embodied and situated AI, In: Proceedings of the International FLAIRS conference, 2004. (online) The Artificial Life Route To Artificial Intelligence: Building Embodied, Situated Agents, Luc Steels and Rodney Brooks Eds., Lawrence Erlbaum Publishing, 1995. (ISBN 978-0805815184) Rodney A. Brooks Cambrian Intelligence (MIT Press, 1999) ISBN 0-262-52263-2; collection of early papers including "Intelligence without representation" and "Intelligence without reason", from 1986 & 1991 respectively. Ronald C. Arkin Behavior-Based Robotics (MIT Press, 1998) ISBN 0-262-01165-4 Hendriks-Jansen, Horst (1996) Catching Ourselves in the Act: Situated Activity, Interactive Emergence, Evolution, and Human Thought. Cambridge, Mass.: MIT Press.
Facebook Platform
The Facebook Platform is the set of services, tools, and products provided by the social networking service Facebook for third-party developers to create their own applications and services that access data in Facebook. The current Facebook Platform was launched in 2010. The platform offers a set of programming interfaces and tools which enable developers to integrate with the open "social graph" of personal relations and other things like songs, places, and Facebook pages. Applications on facebook.com, external websites, and devices are all allowed to access the graph. == History == Facebook launched the Facebook Platform on May 24, 2007, providing a framework for software developers to create applications that interact with core Facebook features. A markup language called Facebook Markup Language was introduced simultaneously; it is used to customize the "look and feel" of applications that developers create. Prior to the Facebook platform, Facebook had built many applications themselves within the Facebook website, including Gifts, allowing users to send virtual gifts to each other, Marketplace, allowing users to post free classified ads, Facebook events, giving users a method of informing their friends about upcoming events, Video, letting users share homemade videos with one another, and social network game, where users can use their connections to friends to help them advance in games they are playing. The Facebook Platform made it possible for outside partners to build similar applications. Many of the popular early social network games would combine capabilities. For instance, one of the early games to reach the top application spot, (Lil) Green Patch, combined virtual Gifts with Event notifications to friends and contributions to charities through Causes. Third-party companies provide application metrics, and several blogs arose in response to the clamor for Facebook applications. On July 4, 2007, Altura Ventures announced the "Altura 1 Facebook Investment Fund," becoming the world's first Facebook-only venture capital firm. On August 29, 2007, Facebook changed the way in which the popularity of applications is measured, to give attention to the more engaging applications, following criticism that ranking applications only by the number of people who had installed the application was giving an advantage to the highly viral, yet useless applications. Tech blog Valleywag has criticized Facebook Applications, labeling them a "cornucopia of uselessness." Others have called for limiting third-party applications so the Facebook user experience is not degraded. Applications that have been created on the Platform include chess, which both allow users to play games with their friends. In such games, a user's moves are saved on the website, allowing the next move to be made at any time rather than immediately after the previous move. By November 3, 2007, seven thousand applications had been developed on the Facebook Platform, with another hundred created every day. By the second annual f8 developers conference on July 23, 2008, the number of applications had grown to 33,000, and the number of registered developers had exceeded 400,000. Within a few months of launching the Facebook Platform, issues arose regarding "application spam", which involves Facebook applications "spamming" users to request it be installed. Facebook integration was announced for the Xbox 360 and Nintendo DSi on June 1, 2009 at E3. On November 18, 2009, Sony announced an integration with Facebook to deliver the first phase of a variety of new features to further connect and enhance the online social experiences of PlayStation 3. On February 2, 2010, Facebook announced the release of HipHop for PHP as an opensource project. Mark Zuckerberg said that his team from Facebook is developing a Facebook search engine. “Facebook is pretty well placed to respond to people’s questions. At some point, we will. We have a team that is working on it", said Mark Zuckerberg. For him, the traditional search engines return too many results that do not necessarily respond to questions. “The search engines really need to evolve a set of answers: 'I have a specific question, answer this question for me.'" On June 10, 2014, Facebook announced Haxl, a Haskell library that simplified the access to remote data, such as databases or web-based services. === Partnerships with device manufacturers === Starting in 2007, Facebook formed data sharing partnerships with at least 60 handset manufacturers, including Apple, Amazon, BlackBerry, Microsoft and Samsung. Those manufacturers were provided with Facebook user data without the users' consent. Most of the partnerships remained in place as of 2018, when the partnerships were first publicly reported. == High-level Platform components == === Graph API === The Graph API is the core of Facebook Platform, enabling developers to read from and write data into Facebook. The Graph API presents a simple, consistent view of the Facebook social graph, uniformly representing objects in the graph (e.g., people, photos, events, and pages) and the connections between them (e.g., friend relationships, shared content, and photo tags). On April 30, 2015, Facebook shut down friends' data API prior to the v2.0 release. === Authentication === Facebook authentication enables developers’ applications to interact with the Graph API on behalf of Facebook users, and it provides a single-sign on mechanism across web, mobile, and desktop apps. ==== Facebook Connect ==== Facebook Connect, also called Log in with Facebook, like OpenID, is a set of authentication APIs from Facebook that developers can use to help their users connect and share with such users' Facebook friends (on and off Facebook) and increase engagement for their website or application. When so used, Facebook members can log on to third-party websites, applications, mobile devices and gaming systems with their Facebook identity and, while logged in, can connect with friends via these media and post information and updates to their Facebook profile. Originally unveiled during Facebook's developer conference, F8, in July 2008, Log in with Facebook became generally available in December 2008. According to an article from The New York Times, "Some say the services are representative of surprising new thinking in Silicon Valley. Instead of trying to hoard information about their users, the Internet companies (including Facebook, Google, MySpace and Twitter) all share at least some of that data so people do not have to enter the same identifying information again and again on different sites." Log in with Facebook cannot be used by users in locations that cannot access Facebook, even if the third-party site is otherwise accessible from that location. According to Facebook, users who logged into The Huffington Post with Facebook spent more time on the site than the average user. === Social plugins === Social plugins – including the Like Button, Recommendations, and Activity Feed – enable developers to provide social experiences to their users with just a few lines of HTML. All social plugins are extensions of Facebook and are designed so that no user data is shared with the sites on which they appear. On the other hand, the social plugins let Facebook track its users’ browsing habits through any sites that feature the plugins. === Open Graph protocol === The Open Graph protocol enables developers to integrate their pages into Facebook's global mapping/tracking tool Social Graph. These pages gain the functionality of other graph objects including profile links and stream updates for connected users. OpenGraph tags in HTML5 might look like this: === iframes === Facebook uses iframes to allow third-party developers to create applications that are hosted separately from Facebook, but operate within a Facebook session and are accessed through a user's profile. Since iframes essentially nest independent websites within a Facebook session, their content is distinct from Facebook formatting. Facebook originally used 'Facebook Markup Language (FBML)' to allow Facebook Application developers to customize the "look and feel" of their applications, to a limited extent. FBML is a specification of how to encode content so that Facebook's servers can read and publish it, which is needed in the Facebook-specific feed so that Facebook's system can properly parse content and publish it as specified. FBML set by any application is cached by Facebook until a subsequent API call replaces it. Facebook also offers a specialized Facebook JavaScript (FBJS) library. Facebook stopped accepting new FBML applications on March 18, 2011, but continued to support existing FBML tabs and applications. Since January 1, 2012 FBML was no longer supported, and FBML no longer functioned as of June 1, 2012. === Microformats === In February 2011, Facebook began to use the hCalendar microformat to mark up events, and the hCard for the events' venues,
Hardware-based encryption
Hardware-based encryption is the use of computer hardware to assist software, or sometimes replace software, in the process of data encryption. Typically, this is implemented as part of the processor's instruction set. For example, the AES encryption algorithm (a modern cipher) can be implemented using the AES instruction set on the ubiquitous x86 architecture. Such instructions also exist on the ARM architecture. However, more unusual systems exist where the cryptography module is separate from the central processor, instead being implemented as a coprocessor, in particular a secure cryptoprocessor or cryptographic accelerator, of which an example is the IBM 4758, or its successor, the IBM 4764. Hardware implementations can be faster and less prone to exploitation than traditional software implementations, and furthermore can be protected against tampering. == History == Prior to the use of computer hardware, cryptography could be performed through various mechanical or electro-mechanical means. An early example is the Scytale used by the Spartans. The Enigma machine was an electro-mechanical system cipher machine notably used by the Germans in World War II. After World War II, purely electronic systems were developed. In 1987 the ABYSS (A Basic Yorktown Security System) project was initiated. The aim of this project was to protect against software piracy. However, the application of computers to cryptography in general dates back to the 1940s and Bletchley Park, where the Colossus computer was used to break the encryption used by German High Command during World War II. The use of computers to encrypt, however, came later. In particular, until the development of the integrated circuit, of which the first was produced in 1960, computers were impractical for encryption, since, in comparison to the portable form factor of the Enigma machine, computers of the era took the space of an entire building. It was only with the development of the microcomputer that computer encryption became feasible, outside of niche applications. The development of the World Wide Web lead to the need for consumers to have access to encryption, as online shopping became prevalent. The key concerns for consumers were security and speed. This led to the eventual inclusion of the key algorithms into processors as a way of both increasing speed and security. == Implementations == === In the instruction set === ==== x86 ==== The X86 architecture, as a CISC (Complex Instruction Set Computer) Architecture, typically implements complex algorithms in hardware. Cryptographic algorithms are no exception. The x86 architecture implements significant components of the AES (Advanced Encryption Standard) algorithm, which can be used by the NSA for Top Secret information. The architecture also includes support for the SHA Hashing Algorithms through the Intel SHA extensions. Whereas AES is a cipher, which is useful for encrypting documents, hashing is used for verification, such as of passwords (see PBKDF2). ==== ARM ==== ARM processors can optionally support Security Extensions. Although ARM is a RISC (Reduced Instruction Set Computer) architecture, there are several optional extensions specified by ARM Holdings. === As a coprocessor === IBM 4758 – The predecessor to the IBM 4764. This includes its own specialised processor, memory and a Random Number Generator. IBM 4764 and IBM 4765, identical except for the connection used. The former uses PCI-X, while the latter uses PCI-e. Both are peripheral devices that plug into the motherboard. === Proliferation === Advanced Micro Devices (AMD) processors are also x86 devices, and have supported the AES instructions since the 2011 Bulldozer processor iteration. Due to the existence of encryption instructions on modern processors provided by both Intel and AMD, the instructions are present on most modern computers. They also exist on many tablets and smartphones due to their implementation in ARM processors. == Advantages == Implementing cryptography in hardware means that part of the processor is dedicated to the task. This can lead to a large increase in speed. In particular, modern processor architectures that support pipelining can often perform other instructions concurrently with the execution of the encryption instruction. Furthermore, hardware can have methods of protecting data from software. Consequently, even if the operating system is compromised, the data may still be secure (see Software Guard Extensions). == Disadvantages == If, however, the hardware implementation is compromised, major issues arise. Malicious software can retrieve the data from the (supposedly) secure hardware – a large class of method used is the timing attack. This is far more problematic to solve than a software bug, even within the operating system. Microsoft regularly deals with security issues through Windows Update. Similarly, regular security updates are released for Mac OS X and Linux, as well as mobile operating systems like iOS, Android, and Windows Phone. However, hardware is a different issue. Sometimes, the issue will be fixable through updates to the processor's microcode (a low level type of software). However, other issues may only be resolvable through replacing the hardware, or a workaround in the operating system which mitigates the performance benefit of the hardware implementation, such as in the Spectre exploit.
Robert Abel and Associates
Robert Abel and Associates (RA&A) was an American pioneering animation production company specializing in television commercials made with computer graphics. Founded by Robert Abel and Con Pederson in 1971, RA&A was especially known for their art direction and won many Clio Awards. Abel and his team created some of the most advanced and impressive computer-animated works of their time, including full ray-traced renders and fluid character animation at a time when such things were largely unknown. A variety of high-profile television advertisements, graphics sequences for motion pictures (including The Andromeda Strain and Tron), and work on laserdisc video games such as Cube Quest, put Abel and his team on the map in the early 1980s. The company was also originally commissioned to create the visual effects for Star Trek: The Motion Picture, but were subsequently taken off the project for mishandling funds. The company was also notable on its work for The Jacksons' 1981 music video "Can You Feel It." RA&A was on the southwest corner of Highland Avenue and Romaine in the heart of Hollywood, California. RA&A closed in 1987 following an ill-fated merger with now-defunct Omnibus Computer Graphics, Inc., a company which had been based in Toronto. Many people who worked at RA&A went on to other ground-breaking projects, including the founding of Wavefront Technologies, Rhythm & Hues and other studios. Many RA&A people went on to win Academy Awards.
Feature engineering
Feature engineering is a preprocessing step in supervised machine learning and statistical modeling which transforms raw data into a more effective set of inputs. Each input comprises several attributes, known as features. By providing models with relevant information, feature engineering significantly enhances their predictive accuracy and decision-making capability. Beyond machine learning, the principles of feature engineering are applied in various scientific fields, including physics. For example, physicists construct dimensionless numbers such as the Reynolds number in fluid dynamics, the Nusselt number in heat transfer, and the Archimedes number in sedimentation. They also develop first approximations of solutions, such as analytical solutions for the strength of materials in mechanics. == Clustering == One of the applications of feature engineering has been clustering of feature-objects or sample-objects in a dataset. Especially, feature engineering based on matrix decomposition has been extensively used for data clustering under non-negativity constraints on the feature coefficients. These include Non-Negative Matrix Factorization (NMF), Non-Negative Matrix-Tri Factorization (NMTF), Non-Negative Tensor Decomposition/Factorization (NTF/NTD), etc. The non-negativity constraints on coefficients of the feature vectors mined by the above-stated algorithms yields a part-based representation, and different factor matrices exhibit natural clustering properties. Several extensions of the above-stated feature engineering methods have been reported in literature, including orthogonality-constrained factorization for hard clustering, and manifold learning to overcome inherent issues with these algorithms. Other classes of feature engineering algorithms include leveraging a common hidden structure across multiple inter-related datasets to obtain a consensus (common) clustering scheme. An example is Multi-view Classification based on Consensus Matrix Decomposition (MCMD), which mines a common clustering scheme across multiple datasets. MCMD is designed to output two types of class labels (scale-variant and scale-invariant clustering), and: is computationally robust to missing information, can obtain shape- and scale-based outliers, and can handle high-dimensional data effectively. Coupled matrix and tensor decompositions are popular in multi-view feature engineering. == Predictive modelling == Feature engineering in machine learning and statistical modeling involves selecting, creating, transforming, and extracting data features. Key components include feature creation from existing data, transforming and imputing missing or invalid features, reducing data dimensionality through methods like Principal Components Analysis (PCA), Independent Component Analysis (ICA), and Linear Discriminant Analysis (LDA), and selecting the most relevant features for model training based on importance scores and correlation matrices. Features vary in significance. Even relatively insignificant features may contribute to a model. Feature selection can reduce the number of features to prevent a model from becoming too specific to the training data set (overfitting). Feature explosion occurs when the number of identified features is too large for effective model estimation or optimization. Common causes include: Feature templates - implementing feature templates instead of coding new features Feature combinations - combinations that cannot be represented by a linear system Feature explosion can be limited via techniques such as regularization, kernel methods, and feature selection. == Automation == Automation of feature engineering is a research topic that dates back to the 1990s. Machine learning software that incorporates automated feature engineering has been commercially available since 2016. Related academic literature can be roughly separated into two types: Multi-relational Decision Tree Learning (MRDTL) uses a supervised algorithm that is similar to a decision tree. Deep Feature Synthesis uses simpler methods. === Multi-relational Decision Tree Learning (MRDTL) === Multi-relational Decision Tree Learning (MRDTL) extends traditional decision tree methods to relational databases, handling complex data relationships across tables. It innovatively uses selection graphs as decision nodes, refined systematically until a specific termination criterion is reached. Most MRDTL studies base implementations on relational databases, which results in many redundant operations. These redundancies can be reduced by using techniques such as tuple id propagation. === Open-source implementations === There are a number of open-source libraries and tools that automate feature engineering on relational data and time series: featuretools is a Python library for transforming time series and relational data into feature matrices for machine learning. MCMD: An open-source feature engineering algorithm for joint clustering of multiple datasets. OneBM or One-Button Machine combines feature transformations and feature selection on relational data with feature selection techniques. OneBM helps data scientists reduce data exploration time allowing them to try and error many ideas in short time. On the other hand, it enables non-experts, who are not familiar with data science, to quickly extract value from their data with a little effort, time, and cost. getML community is an open source tool for automated feature engineering on time series and relational data. It is implemented in C/C++ with a Python interface. It has been shown to be at least 60 times faster than tsflex, tsfresh, tsfel, featuretools or kats. tsfresh is a Python library for feature extraction on time series data. It evaluates the quality of the features using hypothesis testing. tsflex is an open source Python library for extracting features from time series data. Despite being 100% written in Python, it has been shown to be faster and more memory efficient than tsfresh, seglearn or tsfel. seglearn is an extension for multivariate, sequential time series data to the scikit-learn Python library. tsfel is a Python package for feature extraction on time series data. kats is a Python toolkit for analyzing time series data. === Deep feature synthesis === The deep feature synthesis (DFS) algorithm beat 615 of 906 human teams in a competition. == Feature stores == The feature store is where the features are stored and organized for the explicit purpose of being used to either train models (by data scientists) or make predictions (by applications that have a trained model). It is a central location where you can either create or update groups of features created from multiple different data sources, or create and update new datasets from those feature groups for training models or for use in applications that do not want to compute the features but just retrieve them when it needs them to make predictions. A feature store includes the ability to store code used to generate features, apply the code to raw data, and serve those features to models upon request. Useful capabilities include feature versioning and policies governing the circumstances under which features can be used. Feature stores can be standalone software tools or built into machine learning platforms. == Alternatives == Feature engineering can be a time-consuming and error-prone process, as it requires domain expertise and often involves trial and error. Deep learning algorithms may be used to process a large raw dataset without having to resort to feature engineering. However, deep learning algorithms still require careful preprocessing and cleaning of the input data. In addition, choosing the right architecture, hyperparameters, and optimization algorithm for a deep neural network can be a challenging and iterative process.
HtmlUnit
HtmlUnit is a headless web browser written in Java. It allows high-level manipulation of websites from other Java code, including filling and submitting forms and clicking hyperlinks. It also provides access to the structure and the details within received web pages. HtmlUnit emulates parts of browser behaviour including the lower-level aspects of TCP/IP and HTTP. A sequence such as getPage(url), getLinkWith("Click here"), click() allows a user to navigate through hypertext and obtain web pages that include HTML, JavaScript, Ajax and cookies. This headless browser can deal with HTTPS security, basic HTTP authentication, automatic page redirection and other HTTP headers. It allows Java test code to examine returned pages either as text, an XML DOM, or as collections of forms, tables, and links. The goal is to simulate real browsers; namely Chrome, Firefox and Edge. The most common use of HtmlUnit is test automation of web pages, but sometimes it can be used for web scraping, or downloading website content. == Benefits == Provides high-level API, taking away lower-level details away from the user. Compared to other WebDriver implementations, HtmlUnitDriver is the fastest to implement. It can be configured to simulate a specific browser. == Drawbacks == Element layout and rendering can not be tested. The JavaScript support is not complete, which is one of the areas of ongoing enhancements. == Used technologies == W3C DOM HTTP connection, using Apache HttpComponents JavaScript, using forked Rhino HTML Parsing, NekoHTML CSS: using CSS Parser XPath support, using Xalan == Libraries using HtmlUnit == Selenium WebDriver Spring MVC Test Framework Google Web Toolkit tests WebTest Wetator