Psychology of Learning

Unit 2: Operant Conditioning & Choice

Key Terms

Accumulative Record: A graph that shows the cumulative number of responses over time; the slope indicates response rate, with steeper slopes showing higher rates.

Algorithm: A methodical rule that guarantees that a problem will be solved; provides systematic procedures that, if followed correctly, always produce correct solutions.

Anchoring: A bias in decision-making whereby our judgments are influenced by a reference point which is given, even when that reference point is arbitrary & irrelevant.

Area Under the Curve (AUC): An empirical measure of discounting that plots subjective value against delay; larger AUC indicates less steep discounting (more self-control).

Arousal: A general state of physiological & psychological activation ranging from deep sleep to high excitement; affects performance according to the Yerkes-Dodson law.

Automatic Reinforcement: Reinforcement that occurs when the behavior itself is the reinforcing stimulus.

Availability Heuristic: A mental shortcut which makes us believe that things that come easily to mind occur more often than things that do not come easily to mind.

Avoidance: A type of negative reinforcement in which the organism’s response prevents an aversive stimulus from occurring.

Avoidance Learning: Learning to make a response that prevents an aversive stimulus from occurring; maintained by negative reinforcement even though the aversive stimulus is never experienced.

Backward Chaining: A method of teaching a behavioral chain by starting with the last step & working backward; the learner first masters the final response closest to reinforcement.

Baseline: The normal rate of behavior before any experimental manipulation; used as a comparison point for measuring behavior change.

Behavior Modification: The application of learning principles, particularly operant conditioning, to change behavior in applied settings.

Behavioral Contrast: A change in response rate in one component of a multiple schedule that occurs when reinforcement conditions change in the other component.

Behavioral Economics: The study of how organisms allocate their limited resources, including time & money; combines principles from behavioral psychology with microeconomics to predict decision-making.

Bias: In the Generalized Matching Law, a parameter (b) representing a consistent preference for one alternative independent of reinforcement rates.

Bounded Rationality: The concept that human rationality is limited by cognitive constraints including finite information-processing capacity, limited working memory, & time pressures.

Break-and-Run Pattern: A response pattern on fixed-ratio schedules characterized by a pause after reinforcement followed by a high, steady rate of responding until the next reinforcement.

Certainty Effect: The tendency to overweight certain outcomes relative to merely probable outcomes, even when expected values favor the uncertain option.

Chaining: A procedure for establishing a sequence of behaviors in which each response produces a stimulus that serves as both a conditioned reinforcer for the preceding response & a discriminative stimulus for the next response.

Choice: A form of decision-making which involves selecting among alternatives; differs from judgment in requiring selection rather than estimation.

Choice Architecture: The design of environments & contexts in which people make decisions; modifying choice architecture is more effective than relying on willpower.

Closed-Loop Movement: Movement that relies on continuous sensory feedback for guidance & correction; typically slower movements that can be adjusted during execution.

Compensatory Strategies: Decision-making strategies that involve trading low values on one dimension for high values on another; weaknesses on some dimensions can be offset by strengths on others.

Concurrent Schedule: A reinforcement schedule in which two or more schedules are available simultaneously, each associated with a different response.

Continuous Motor Skills: Motor skills with no distinct beginning or end point, such as swimming, running, or steering; require ongoing adjustments throughout performance.

Continuous Reinforcement (CRF): A schedule in which every correct response is reinforced; also called FR-1.

Cumulative Recorder: A device that produces a continuous record of responding over time by moving paper at a constant rate while a pen moves upward with each response.

Decision Tree: A visual tool that maps out all possible choices, outcomes, & their consequences, helping decision-makers systematically evaluate options.

Decision-Making Under Certainty: Decisions made in which the factors that determine the outcomes of the different choice alternatives are known with complete accuracy.

Decision-Making Under Risk: Decisions made in which the factors that determine the outcomes of the different choice alternatives occur with known probabilities.

Decision-Making Under Uncertainty: Decisions made in which the probabilities of the different factors which affect the outcomes of the different choice alternatives are not known with precision or cannot be determined.

Delay Discounting: The decrease in the subjective value of a reward as the delay to receiving that reward increases; also called temporal discounting.

Descriptive Models of Decision-Making: Models that address how humans actually make decisions, describing observed behavior rather than prescribing optimal behavior.

Differential Reinforcement: Reinforcing responses that meet a specific criterion while withholding reinforcement for responses that do not meet the criterion.

Discrete Motor Skills: Motor skills with a recognizable beginning & end point, such as throwing a ball or pressing a button; can be performed as single units.

Discriminative Stimulus (S^D): A stimulus that signals the availability of reinforcement for a particular response; sets the occasion for responding.

Distributed Practice: Practice sessions that are spaced apart with rest intervals between them; generally produces better long-term retention than massed practice.

Drive Reduction Theory: Hull’s theory that reinforcement works by reducing biological drives; problematic because some reinforcers increase arousal rather than reduce drives.

Ego Depletion: The state in which willpower has been temporarily depleted through prior self-control efforts, making subsequent self-control attempts more likely to fail.

Elastic Demand: In behavioral economics, consumption that decreases substantially when price increases; indicates the reinforcer has available substitutes.

Electrical Stimulation of the Brain (ESB): Direct stimulation of brain regions that can serve as a powerful reinforcer; rats will cross electrified floors or forgo food to obtain ESB.

Escape: A type of negative reinforcement in which the organism’s response terminates an ongoing aversive stimulus.

Expected Utility: Calculated by multiplying the probability of each outcome times the utility (value) of that outcome, then summing across all outcomes.

External Imagery: Mental imagery from a third-person perspective, as if watching yourself perform on video; emphasizes visual form & technique.

Extinction: The procedure of withholding reinforcement for a previously reinforced response, resulting in a decrease in the response.

Extinction Burst: A temporary increase in the frequency, duration, or intensity of a response when extinction is first implemented.

Fast Strategy: A life history strategy adaptive in high mortality, unstable environments characterized by early reproduction, steep delay discounting, & high risk-taking.

Fast-and-Frugal Trees: Shortcuts for decision-making involving a series of yes/no questions to come to a decision very quickly; use minimal information for sequential binary decisions.

Fixed Action Pattern: An innate, stereotyped behavior sequence triggered by a specific stimulus; can interfere with operant conditioning when tokens trigger instinctive behaviors.

Fixed Interval (FI): A schedule in which reinforcement is delivered for the first response after a fixed period of time has elapsed since the last reinforcement.

Fixed Ratio (FR): A schedule in which reinforcement is delivered after a fixed number of responses.

Forward Chaining: A method of teaching a behavioral chain by starting with the first step & proceeding to later steps; the learner first masters the initial response in the sequence.

Framing: A bias in decision-making whereby our decisions are influenced by the way a question is asked or a choice is presented.

Future-Uncertainty: Uncertainty about the long-term consequences of behavior; one of two types of uncertainty (along with present-uncertainty) that reduce prudent decision-making.

Gambler’s Fallacy: A bias in judgment whereby our estimations of probability are heavily influenced by the most recent outcomes, leading us to believe past outcomes affect future independent events.

Generalized Matching Law: Baum’s extension of the Matching Law that incorporates bias & sensitivity parameters to account for systematic deviations from perfect matching.

Heuristic: A mental shortcut that helps decision-makers make judgments & choices quickly in the face of uncertainty; works from partial information.

Hyperbolic Equation: The mathematical function V = A / (1 + kD) that best describes delay discounting, where V = discounted value, A = actual value, k = discount rate, & D = delay.

Impulsivity: Preference for smaller-immediate rewards over larger-delayed rewards; demonstrated in paradigms where organisms choose immediate but smaller reinforcement.

Indifference Points: Combinations of options where someone is equally likely to choose either alternative; reveal how steeply a person discounts future rewards.

Inelastic Demand: In behavioral economics, consumption that remains relatively constant despite price increases; indicates the reinforcer has few substitutes or is essential.

Information Cascade: A situation where sequential decision-makers ignore their own information & copy predecessors, potentially propagating errors through populations.

Information Overload: A state where too much information is available, exceeding cognitive capacity to process it effectively, leading to poorer decisions.

Instinctive Drift: The tendency for learned behaviors to drift toward instinctive behaviors over time; demonstrated by Breland & Breland with raccoons & pigs.

Intermittent Reinforcement: A schedule in which only some responses are reinforced; also called partial reinforcement.

Internal Imagery: Mental imagery from a first-person perspective, experiencing the kinesthetic sensations of performing the movement; emphasizes feeling the body move.

Intuition: A type of thinking that involves understanding that is quick & effortless & often involves insight; generally, people cannot verbalize the thought processes.

Inverted-U Theory: The theory that performance is optimal at moderate levels of arousal & deteriorates at both low & high arousal levels; also called the Yerkes-Dodson law.

Ironic Errors Theory: Wegner’s theory that trying not to make a specific error can increase the likelihood of making that error, especially under pressure or cognitive load.

Judgment: A type of decision-making in which one estimates the likelihood of an event; examples include estimating probability of rain or assessing chances of success.

Knowledge of Performance (KP): Feedback about the movement pattern or technique used to produce a response; information about how the movement was executed.

Knowledge of Results (KR): Feedback about the outcome of a movement in terms of the environmental goal; information about whether the goal was achieved.

Latent Learning: Learning that occurs without obvious reinforcement & is not demonstrated until reinforcement is introduced; demonstrated by Tolman & Honzik with rats in mazes.

Law of Diminishing Marginal Value: The principle that each additional unit of a commodity provides less additional utility than the previous unit; explains why variety in choices can maximize total utility.

Life History Theory (LHT): An analytical framework designed to study the diversity of life history strategies; predicts how organisms should allocate limited resources among competing demands.

Local Reinforcement Rate: The reinforcement rate currently being received from a particular alternative, as opposed to the overall or average rate.

Loss Aversion: The phenomenon where people are more motivated to avoid losses than to acquire equivalent gains; a loss of $1000 is more painful than a gain of $1000 is joyful.

Massed Practice: Practice sessions that are continuous with little or no rest between trials; can lead to fatigue & poorer long-term retention.

Matching Law: The principle that in a two-choice situation, the percentage of responses made on each alternative will match the percentage of reinforcers received from each alternative.

Maximum Utility Theory: The economic principle that rational consumers make choices to maximize total utility or satisfaction from their resources.

Melioration: A theory proposing that organisms shift behavior toward whichever alternative provides better local reinforcement rates, even when this doesn’t maximize global outcomes.

Mental Imagery: Cognitive rehearsal of a movement without physical execution; involves activation of motor areas in the brain & can improve performance.

Momentary Maximization Theory: A theory of choice that argues organisms make choices to maximize their satisfaction at the present moment rather than considering long-term consequences.

Motor Programs: Pre-structured sets of motor commands that can be executed without continuous feedback; allow rapid, automatic movement sequences.

Motor Trace: In Adams’s theory, the memory representation responsible for initiating movement; selects & initiates the motor commands.

Movement Sequences: Series of movements performed in a fixed order with specific timing; can be cyclical (walking) or non-cyclical (typing a word).

Mowrer’s Two-Factor Theory: The theory that avoidance learning involves two processes: classical conditioning creates fear of warning stimuli, & operant conditioning maintains avoidance responses that reduce fear.

Multiple Schedule: A schedule in which two or more schedules alternate, each associated with a different discriminative stimulus.

Need Reduction Theory: The theory that reinforcement works by satisfying biological needs; problematic because saccharin reinforces without satisfying nutritional needs.

Negative Punishment: The removal of a stimulus following a response that decreases the future probability of that response; also called omission training.

Negative Reinforcement: The removal of an aversive stimulus following a response that increases the future probability of that response.

Negative Transfer: When prior learning interferes with new learning; occurs when old & new tasks require different responses to similar stimuli.

Normative Models of Decision-Making: Models that address how humans should make decisions to be rational & maximize expected outcomes; establish standards for optimal decision-making.

Open-Loop Movement: Movement that is executed without ongoing sensory feedback; typically rapid, ballistic movements that cannot be corrected once initiated.

Operant Behavior: Behavior that operates on the environment to produce consequences; behavior that is controlled by its consequences.

Operant Conditioning: A type of learning in which behavior is modified by its consequences; responses that produce favorable consequences are strengthened while those that produce unfavorable consequences are weakened.

Optimal Foraging: Behavior patterns in which animals make choices that maximize their net resource intake per unit time; involves balancing current site quality against search costs.

Optimization Theory: A theory of choice behavior which assumes that consumers spend their resources in ways that maximize their utility.

Overconfidence: A bias in decision-making whereby we are more confident than correct; an overestimation of the accuracy of one’s beliefs & judgments.

Overmatching: A deviation from matching in which animals show a stronger preference for the better alternative than matching predicts; occurs when sensitivity parameter a > 1.

Partial Reinforcement Effect (PRE): The finding that behaviors reinforced on intermittent schedules are more resistant to extinction than behaviors reinforced continuously.

Perceptual Trace: In Adams’s theory, the memory representation of what correct performance feels like; serves as a reference standard for error detection.

Positive Punishment: The presentation of an aversive stimulus following a response that decreases the future probability of that response.

Positive Reinforcement: The presentation of a stimulus following a response that increases the future probability of that response.

Positive Transfer: When prior learning facilitates new learning; occurs when old & new tasks share similar stimulus-response relationships.

Post-Reinforcement Pause (PRP): A pause in responding that occurs after reinforcement delivery, particularly on fixed schedules.

Precommitment: A situation where the individual makes a decision well before the actual time where consequences would be given & removes the ability to change that decision later.

Preference Reversals: Changes in what a person prefers as time passes, even though objective consequences remain constant; explains why people plan well but execute poorly.

Premack Principle: The principle that high-probability behaviors can reinforce low-probability behaviors; a more preferred activity can serve as a reinforcer for a less preferred activity.

Present-Uncertainty: Uncertainty about the immediate consequences of behavior; one of two types of uncertainty (along with future-uncertainty) that reduce prudent decision-making.

Prestige Bias: The tendency to copy choices of successful, high-status individuals.

Probability Discounting: The loss of value of a reinforcer that is to be received only probabilistically; a certain reward is worth more than an uncertain reward of equal objective value.

Probability of Reciprocation: The subjective likelihood that one’s cooperative behavior will be matched by another (in social cooperation) or that present sacrifice will yield future benefit (in self-control); the crucial variable distinguishing self-control from social cooperation dilemmas.

Probability Theory: A branch of mathematics that deals with random events & allows for the best average prediction to be made; provides formal methods for reasoning about uncertain outcomes.

Progressive Ratio Schedule: A schedule in which the ratio requirement increases systematically after each reinforcement; used to assess motivation or reinforcer efficacy.

Prospect Theory: A descriptive theory of choice under risk showing that people evaluate options relative to reference points & weight losses more heavily than equivalent gains.

Prudent Decision-Making: Decision-making driven by long-term consequences rather than immediate outcomes; essentially synonymous with self-control in the context of repeated choices.

Pseudocertainty Effect: Treating conditional certainty (certainty within an uncertain scenario) as if it were absolute certainty.

Punishment: A consequence that decreases the future probability of the behavior it follows.

Rapid Adjusting Procedure (Titration): A procedure that systematically adjusts the value of one option based on previous choices until an indifference point is found.

Recognition Heuristic: A mental shortcut that biases us towards choosing those things that are familiar to us, assuming familiarity indicates quality or importance.

Reflection Effect: The reversal of risk preferences between gain & loss domains—risk aversion for gains but risk seeking for losses.

Regret Theory: A descriptive model of decision-making that assumes decision-makers experience feelings of regret & try to avoid this feeling when making decisions.

Reinforcement: A consequence that increases the future probability of the behavior it follows.

Reinforcement Relativity: The principle that whether an activity serves as a reinforcer depends on its probability relative to other available activities, not on absolute properties.

Reinforcement Schedule: A rule specifying which responses will be reinforced; determines the pattern & timing of reinforcement delivery.

Representativeness Heuristic: A mental shortcut used to assess probability whereby the more an object appears to represent a class of objects, the more likely we judge it to belong to that class.

Response Chain: A sequence of behaviors in which each response produces stimuli that serve as discriminative stimuli for the next response in the chain.

Response Deprivation Theory: The theory that restricting access to any activity below its baseline level creates conditions for that activity to serve as a reinforcer.

Reward Deficiency Syndrome: A theory suggesting that some individuals have underactive reward systems, leading them to seek excessive stimulation through addictive behaviors.

Satisficing: A theory of decision-making arguing that decision-makers work to satisfy basic needs & achieve acceptable outcomes rather than searching exhaustively for the absolute best option.

Scallop Pattern: A response pattern on fixed-interval schedules characterized by a pause after reinforcement followed by gradually accelerating responding as the interval elapses.

Schema Theory: Schmidt’s theory that motor learning involves developing generalized rules (schemas) relating movement parameters to outcomes, enabling transfer to novel movements.

Self-Control: Control of behavior using internal controls rather than external controls; involves choosing larger-later rewards over smaller-sooner rewards.

Sensitivity: In the Generalized Matching Law, a parameter (a) representing how responsive the organism is to differences in reinforcement rates.

Shaping: The reinforcement of successive approximations to a target behavior; used to establish behaviors not currently in the organism’s repertoire.

Slow Strategy: A life history strategy adaptive in low mortality, stable environments characterized by delayed reproduction, shallow delay discounting, & risk aversion.

Social Cooperation Problems: Decision situations where one choice is easy (often selfish) & another is harder but offers superior long-term consequences for the community.

Social Discounting: The phenomenon where people care less about socially distant others than about themselves & close others; correlated with temporal discounting.

Social Learning: Acquiring information about options, outcomes, & strategies by observing others rather than through personal direct experience.

Social Proof: The tendency to infer that popular choices are good choices based on others’ behavior.

Spontaneous Recovery: The reappearance of an extinguished response after a period of time following extinction.

Stochastic Models of Choice: Models that treat preferences as though they have a random component, allowing understanding of how a person can prefer different options on different occasions.

Subjective Utility Theory: An extension of expected utility theory used when probabilities cannot be determined in advance; decision-makers estimate subjective probabilities based on beliefs & experience.

Sunk Cost Fallacy: The tendency to continue investing in something because of past investments that cannot be recovered, rather than basing decisions solely on future costs & benefits.

Sunk Costs: Past investments (time, money, effort) that have already been incurred & cannot be recovered regardless of future decisions.

Superstitious Behavior: Behavior that is accidentally reinforced by coincidental pairing with reinforcement, though there is no causal relationship between the behavior & the reinforcer.

Take-the-Best Heuristic: A mental shortcut based on a single good reason, ignoring all other information; identify the most important cue & make your decision based on that cue alone.

Three-Term Contingency: The basic unit of operant analysis consisting of the discriminative stimulus (S^D), the response (R), & the consequence (S^R); also called the ABCs of behavior.

Time Horizons: Points in time in the future beyond which we do not understand or plan; the ultimate form of delay discounting where consequences fail to be represented.

Tragedy of the Commons: A conflict where each individual, seeking their own selfish best interest, utilizes common resources in a way that is ultimately destructive to the community as a whole.

Transfer of Training: The influence of prior learning on new learning; can be positive (facilitating) or negative (interfering).

Type A Personality: A personality pattern characterized by competitiveness, time urgency, hostility, & achievement striving; associated with maximizing decision strategies.

Type B Personality: A personality pattern characterized by a relaxed, patient, & easygoing approach; associated with satisficing decision strategies.

Undermatching: A deviation from matching in which animals express relative indifference to the alternatives in a two-choice procedure; occurs when sensitivity parameter a < 1.

Universal Human Morality: A human condition described by C.S. Lewis shared by all individuals; an internal control which helps keep selfish behavior in check.

Utilitarianism: A theory of morality that argues the best choice is always the one which maximizes the utility of all people that will be affected by the choice.

Utility: A measure used in economics to describe the subjective satisfaction, value, or benefit gained from consuming a good or service; in psychology, represents reinforcing value.

Variable Interval (VI): A schedule in which reinforcement is delivered for the first response after a variable period of time has elapsed, averaging a specified duration.

Variable Ratio (VR): A schedule in which reinforcement is delivered after a variable number of responses, averaging a specified number.

Willpower: The cognitive resource required for self-regulation; like a muscle that gets fatigued with use but can be strengthened through regular exercise.

Yerkes-Dodson Law: The principle that optimal performance occurs at moderate arousal levels, with the optimal level varying based on task complexity.