Table of Contents

Preface xv

1 Introduction 1

1.1 Introduction to System Identification 1

1.1.1 System Models and Simulation 1

1.1.2 Systems and Signals 3

1.1.3 System Identification 3

1.2 Linear System Identification 3

1.3 Nonlinear System Identification 5

1.4 NARMAX Methods 7

1.5 The NARMAX Philosophy 8

1.6 What is System Identification For? 9

1.7 Frequency Response of Nonlinear Systems 11

1.8 Continuous-Time, Severely Nonlinear, and Time-Varying Models and Systems 12

1.9 Spatio-temporal Systems 13

1.10 Using Nonlinear System Identification in Practice and Case Study Examples 13

References 14

2 Models for Linear and Nonlinear Systems 17

2.1 Introduction 17

2.2 Linear Models 18

2.2.1 Autoregressive Moving Average with Exogenous Input Model 18

2.2.2 Parameter Estimation for Linear Models 20

2.3 Piecewise Linear Models 22

2.3.1 Spatial Piecewise Linear Models 23

2.3.2 Models with Signal-Dependent Parameters 26

2.3.3 Remarks on Piecewise Linear Models 29

2.4 Volterra Series Models 30

2.5 Block-Structured Models 31

2.5.1 Parallel Cascade Models 32

2.5.2 Feedback Block-Structured Models 32

2.6 NARMAX Models 33

2.6.1 Polynomial NARMAX Model 35

2.6.2 Rational NARMAX Model 37

2.6.3 The Extended Model Set Representation 39

2.7 Generalised Additive Models 40

2.8 Neural Networks 41

2.8.1 Multi-layer Networks 41

2.8.2 Single-Layer Networks 42

2.9 Wavelet Models 45

2.9.1 Dynamic Wavelet Models 46

2.10 State-Space Models 48

2.11 Extensions to the MIMO Case 49

2.12 Noise Modelling 49

2.12.1 Noise-Free 50

2.12.2 Additive Random Noise 50

2.12.3 Additive Coloured Noise 50

2.12.4 General Noise 51

2.13 Spatio-temporal Models 52

References 53

3 Model Structure Detection and Parameter Estimation 61

3.1 Introduction 61

3.2 The Orthogonal Least Squares Estimator and the Error Reduction Ratio 64

3.2.1 Linear-in-the-Parameters Representation 64

3.2.2 The Matrix Form of the Linear-in-the-Parameters Representation 65

3.2.3 The Basic OLS Estimator 65

3.2.4 The Matrix Formulation of the OLS Estimator 67

3.2.5 The Error Reduction Ratio 68

3.2.6 An Illustrative Example of the Basic OLS Estimator 69

3.3 The Forward Regression OLS Algorithm 70

3.3.1 Forward Regression with OLS 72

3.3.2 An Illustrative Example of Forward Regression with OLS 77

3.3.3 The OLS Estimation Engine and Identification Procedure 78

3.4 Term and Variable Selection 79

3.5 OLS and Sum of Error Reduction Ratios 80

3.5.1 Sum of Error Reduction Ratios 82

3.5.2 The Variance of the s -Step-Ahead Prediction Error 82

3.5.3 The Final Prediction Error 83

3.5.4 The Variable Selection Algorithm 83

3.6 Noise Model Identification 84

3.6.1 The Noise Model 84

3.6.2 A Simulation Example with Noise Modelling 87

3.7 An Example of Variable and Term Selection for a Real Data Set 87

3.8 ERR is Not Affected by Noise 94

3.9 Common Structured Models to Accommodate Different Parameters 95

3.10 Model Parameters as a Function of Another Variable 98

3.10.1 System Internal and External Parameters 98

3.10.2 Parameter-Dependent Model Structure 98

3.10.3 Modelling Auxetic Foams – An Example of External Parameter-Dependent Model Identification 99

3.11 OLS and Model Reduction 100

3.12 Recursive Versions of OLS 102

References 102

4 Feature Selection and Ranking 105

4.1 Introduction 105

4.2 Feature Selection and Feature Extraction 106

4.3 Principal Components Analysis 107

4.4 A Forward Orthogonal Search Algorithm 108

4.4.1 The Basic Idea of the FOS-MOD Algorithm 108

4.4.2 Feature Detection and Ranking 109

4.4.3 Monitoring the Search Procedure 111

4.4.4 Illustrative Examples 112

4.5 A Basis Ranking Algorithm Based on PCA 113

4.5.1 Principal Component-Derived Multiple Regression 113

4.5.2 PCA-Based MFROLS Algorithms 114

4.5.3 An Illustrative Example 115

References 117

5 Model Validation 119

5.1 Introduction 119

5.2 Detection of Nonlinearity 121

5.3 Estimation and Test Data Sets 123

5.4 Model Predictions 124

5.4.1 One-Step-Ahead Prediction 124

5.4.2 Model Predicted Output 126

5.5 Statistical Validation 127

5.5.1 Correlation Tests for Input–Output Models 128

5.5.2 Correlation Tests for Time Series Models 132

5.5.3 Correlation Tests for MIMO Models 133

5.5.4 Output-Based Tests 134

5.6 Term Clustering 135

5.7 Qualitative Validation of Nonlinear Dynamic Models 137

5.7.1 Poincaré Sections 139

5.7.2 Bifurcation Diagrams 139

5.7.3 Cell Maps 140

5.7.4 Qualitative Validation in Nonlinear System Identification 140

References 145

6 The Identification and Analysis of Nonlinear Systems in the Frequency Domain 149

6.1 Introduction 149

6.2 Generalised Frequency Response Functions 151

6.2.1 The Volterra Series Representation of Nonlinear Systems 153

6.2.2 Generalised Frequency Response Functions 156

6.2.3 The Relationship Between GFRFs and Output Response of Nonlinear Systems 157

6.2.4 Interpretation of the Composition of the Output Frequency Response of Nonlinear Systems 162

6.2.5 Estimation and Computation of GFRFs 165

6.2.6 The Analysis of Nonlinear Systems Using GFRFs 176

6.3 Output Frequencies of Nonlinear Systems 184

6.3.1 Output Frequencies of Nonlinear Systems under Multi-tone Inputs 185

6.3.2 Output Frequencies of Nonlinear Systems for General Inputs 187

6.4 Nonlinear Output Frequency Response Functions 191

6.4.1 Definition and Properties of NOFRFs 192

6.4.2 Evaluation of NOFRFs 195

6.4.3 Damage Detection Using NARMAX Modelling and NOFRF-Based Analysis 196

6.5 Output Frequency Response Function of Nonlinear Systems 202

6.5.1 Definition of the OFRF 203

6.5.2 Determination of the OFRF 203

6.5.3 Application of the OFRF to Analysis of Nonlinear Damping for Vibration Control 207

References 213

7 Design of Nonlinear Systems in the Frequency Domain – Energy Transfer Filters and Nonlinear Damping 217

7.1 Introduction 217

7.2 Energy Transfer Filters 218

7.2.1 The Time and Frequency Domain Representation of the NARX Model with Input Nonlinearity 220

7.2.2 Energy Transfer Filter Designs 222

7.3 Energy Focus Filters 240

7.3.1 Output Frequencies of Nonlinear Systems with Input Signal Energy Located in Two Separate Frequency Intervals 241

7.3.2 The Energy Focus Filter Design Procedure and an Example 245

7.4 OFRF-Based Approach for the Design of Nonlinear Systems in the Frequency Domain 249

7.4.1 OFRF-Based Design of Nonlinear Systems in the Frequency Domain 249

7.4.2 Design of Nonlinear Damping in the Frequency Domain for Vibration Isolation: An Experimental Study 251

References 259

8 Neural Networks for Nonlinear System Identification 261

8.1 Introduction 261

8.2 The Multi-layered Perceptron 263

8.3 Radial Basis Function Networks 264

8.3.1 Training Schemes for RBF Networks 266

8.3.2 Fixed Kernel Centres with a Single Width 266

8.3.3 Limitation of RBF Networks with a Single Kernel Width 268

8.3.4 Fixed Kernel Centres and Multiple Kernel Widths 269

8.4 Wavelet Networks 270

8.4.1 Wavelet Decompositions 271

8.4.2 Wavelet Networks 272

8.4.3 Limitations of Fixed Grid Wavelet Networks 273

8.4.4 A New Class of Wavelet Networks 274

8.5 Multi-resolution Wavelet Models and Networks 277

8.5.1 Multi-resolution Wavelet Decompositions 277

8.5.2 Multi-resolution Wavelet Models and Networks 280

8.5.3 An Illustrative Example 282

References 284

9 Severely Nonlinear Systems 289

9.1 Introduction 289

9.2 Wavelet NARMAX Models 291

9.2.1 Nonlinear System Identification Using Wavelet Multi-resolution NARMAX Models 292

9.2.2 A Strategy for Identifying Nonlinear Systems 299

9.3 Systems that Exhibit Sub-harmonics and Chaos 301

9.3.1 Limitations of the Volterra Series Representation 301

9.3.2 Time Domain Analysis 302

9.4 The Response Spectrum Map 305

9.4.1 Introduction 305

9.4.2 Examples of the Response Spectrum Map 306

9.5 A Modelling Framework for Sub-harmonic and Severely Nonlinear Systems 313

9.5.1 Input Signal Decomposition 314

9.5.2 MISO NARX Modelling in the Time Domain 317

9.6 Frequency Response Functions for Sub-harmonic Systems 320

9.6.1 MISO Frequency Domain Volterra Representation 320

9.6.2 Generating the GFRFs from the MISO Model 322

9.7 Analysis of Sub-harmonic Systems and the Cascade to Chaos 326

9.7.1 Frequency Domain Response Synthesis 326

9.7.2 An Example of Frequency Domain Analysis for Sub-harmonic Systems 332

References 334

10 Identification of Continuous-Time Nonlinear Models 337

10.1 Introduction 337

10.2 The Kernel Invariance Method 338

10.2.1 Definitions 338

10.2.2 Reconstructing the Linear Model Terms 342

10.2.3 Reconstructing the Quadratic Model Terms 346

10.2.4 Model Structure Determination 348

10.3 Using the GFRFs to Reconstruct Nonlinear Integro-differential Equation Models Without Differentiation 352

10.3.1 Introduction 352

10.3.2 Reconstructing the Linear Model Terms 355

10.3.3 Reconstructing the Quadratic Model Terms 358

10.3.4 Reconstructing the Higher-Order Model Terms 361

10.3.5 A Real Application 364

References 367

11 Time-Varying and Nonlinear System Identification 371

11.1 Introduction 371

11.2 Adaptive Parameter Estimation Algorithms 372

11.2.1 The Kalman Filter Algorithm 372

11.2.2 The RLS and LMS Algorithms 375

11.2.3 Some Practical Considerations for the KF, RLS, and LMS Algorithms 376

11.3 Tracking Rapid Parameter Variations Using Wavelets 376

11.3.1 A General Form of TV-ARX Models Using Wavelets 376

11.3.2 A Multi-wavelet Approach for Time-Varying Parameter Estimation 377

11.4 Time-Dependent Spectral Characterisation 378

11.4.1 The Definition of a Time-Dependent Spectral Function 378

11.5 Nonlinear Time-Varying Model Estimation 380

11.6 Mapping and Tracking in the Frequency Domain 381

11.6.1 Time-Varying Frequency Response Functions 381

11.6.2 First and Second-Order TV-GFRFs 382

11.7 A Sliding Window Approach 388

References 389

12 Identification of Cellular Automata and N -State Models of Spatio-temporal Systems 391

12.1 Introduction 391

12.2 Cellular Automata 393

12.2.1 History of Cellular Automata 393

12.2.2 Discrete Lattice 393

12.2.3 Neighbourhood 394

12.2.4 Transition Rules 396

12.2.5 Simulation Examples of Cellular Automata 399

12.3 Identification of Cellular Automata 402

12.3.1 Introduction and Review 402

12.3.2 Polynomial Representation 403

12.3.3 Neighbourhood Detection and Rule Identification 405

12.4 N -State Systems 414

12.4.1 Introduction to Excitable Media Systems 414

12.4.2 Simulation of Excitable Media 415

12.4.3 Identification of Excitable Media Using a CA Model 419

12.4.4 General N-State Systems 424

References 427

13 Identification of Coupled Map Lattice and Partial Differential Equations of Spatio-temporal Systems 431

13.1 Introduction 431

13.2 Spatio-temporal Patterns and Continuous-State Models 432

13.2.1 Stem Cell Colonies 433

13.2.2 The Belousov–Zhabotinsky Reaction 434

13.2.3 Oxygenation in Brain 434

13.2.4 Growth Patterns 435

13.2.5 A Simulated Example Showing Spatio-temporal Chaos from CML Models 435

13.3 Identification of Coupled Map Lattice Models 437

13.3.1 Deterministic CML Models 437

13.3.2 The Identification of Stochastic CML Models 454

13.4 Identification of Partial Differential Equation Models 458

13.4.1 Model Structure 458

13.4.2 Time Discretisation 459

13.4.3 Nonlinear Function Approximation 459

13.5 Nonlinear Frequency Response Functions for Spatio-temporal Systems 466

13.5.1 A One-Dimensional Example 467

13.5.2 Higher-Order Frequency Response Functions 468

References 471

14 Case Studies 473

14.1 Introduction 473

14.2 Practical System Identification 474

14.3 Characterisation of Robot Behaviour 478

14.3.1 Door Traversal 478

14.3.2 Route Learning 482

14.4 System Identification for Space Weather and the Magnetosphere 484

14.5 Detecting and Tracking Iceberg Calving in Greenland 493

14.5.1 Causality Detection 494

14.5.2 Results 495

14.6 Detecting and Tracking Time-Varying Causality for EEG Data 498

14.6.1 Data Acquisition 499

14.6.2 Causality Detection 500

14.6.3 Detecting Linearity and Nonlinearity 504

14.7 The Identification and Analysis of Fly Photoreceptors 505

14.7.1 Identification of the Fly Photoreceptor 506

14.7.2 Model-Based System Analysis in the Time and Frequency Domain 507

14.8 Real-Time Diffuse Optical Tomography Using RBF Reduced-Order Models of the Propagation of Light for Monitoring Brain Haemodynamics 514

14.8.1 Diffuse Optical Imaging 515

14.8.2 In-vivo Real-Time 3-D Brain Imaging Using Reduced-Order Forward Models 517

14.9 Identification of Hysteresis Effects in Metal Rubber Damping Devices 522

14.9.1 Dynamic Modelling of Metal Rubber Damping Devices 523

14.9.2 Model Identification of a Metal Rubber Specimen 526

14.10 Identification of the Belousov–Zhabotinsky Reaction 528

14.10.1 Data Acquisition 529

14.10.2 Model Identification 530

14.11 Dynamic Modelling of Synthetic Bioparts 534

14.11.1 The Biopart and the Experiments 535

14.11.2 NARMAX Model of the Synthetic Biopart 536

14.12 Forecasting High Tides in the Venice Lagoon 539

14.12.1 Time Series Forecasting Problem 540

14.12.2 Water-Level Modelling and High-Tide Forecasting 541

References 543

Index 549

About the Author

Stephen A. Billings is Professor of Signal Processing and Complex Systems, and Director of the Signal Processing and Complex Systems Research Group, in the Department of Automatic Control and Systems Engineering at the University of Sheffield, He is counted as "highly cited" by the ISI Web of Knowledge with 250 publications to his name.