Wandering through the multi-dimensional abyss

I have not posted to this blog for a long time partly because I lost my focus on the Numenta program within the Machine Understanding arena, and partly because I have been distracted by the need to earn money, and even the political silly season. On a good note my friend Dan Hamburg got elected to be Mendocino County Supervisor, and the government of California can now approve budgets with a majority vote. Which does not mean they will start doing budgets on time, or balance them, or run the state well; but they could.

I found, slogging through Judea Pearl's Probabilistic Reasoning in Intelligent Systems
and Terrence Fine's Probability and Probabilistic Reasoning for Electrical Engineering
, that my mind kept fuzzing up. Maybe I am slowing down in my old age, but the real problem was that my last formal training in probability was when I was 19, and interpreting clinical trial p values is guestimate work. I had to regress to a simpler text than what I used in college, and I can recommend Finite Mathematics with Applications
by A. W. Goodman and J.S. Ratti for introductions to simple probability, conditional probability, Bayes' Theorem, and even Markov Chains that were simple enough for me to feel I really understood easy examples and the concepts themselves.

But my wanderings have been further afield than that. I continue to be fascinated with tensors, and got a lot out of Introduction to Tensor Calculus, Relativity and Cosmology
by D. F. Lawdon. Again, I never got to tensors in college (I ended up a Political Science major), and thinking I was brighter than I really was (brightness is mainly a function of prepartion, I now know), started off with mathematical treatments that were too abstract for me to do more than pretend to follow.

I have even got stuck on Maxwell's equations for electromagnetism. Now we all should admit that if we read broadly in math and science we don't take the time to really understand everything; we trust our fellows to have done their homework before a set of facts or an equation is presented in a paper or textbook. We may like to feel we agree with quantum physics, but who except for professional physicists have the time to really look at the data and the math in detail? I have always assumed that Maxwell's equations are correct, and that if I needed to I could look up the definitions of curl, etc., and do the math. But that is not the same thing as the deep understanding one gets from working in electromagnetics on a regular basis.

I have wandered farther afield than that, to Lie groups and Galois theory, which may have nothing to do with machine understanding. Nevertheless, I wander. And I keep coming back to what is known about the structure of the cortex, of the actual tangles of nerve cells themselves, and in particular to the way pyramidal cells span multiple layers of the cortex with their intricate axons and dendrites. How do you create a math that represents such a tangle? Skipping that, you can do funtional units as Numenta does, or you can try the AI tradition with its tradition trying to get the end results without understanding the details of how neurons actually get stuff done.

Right now I have little paid work going on, so I may be writing in the blog more often. If paid work becomes available, there will be more delays.

Understanding Probability and Probabilistic Reasoning

I think I mentioned earlier that I have had problems with the Numenta model as described in "Towards a Mathematical Theory of Cortical Micro-circuits" because of the use of probability-based mathematics. It seems to me that neurons are deterministic mechanisms. But I have noted in the past that I can be pretty dim-witted at times, and decided to study Numenta's HTM systems anyway. I reminded myself that quantum physics has two different formulations, one based on matrix algebra and the other on the Schrodinger equation. They both work, and some brilliant person showed that they are formally the same long, long ago. So when thinking about or solving problems you can use whichever is easiest or gives the best insights. The same way some physics and math problems are easier in polar coordinates than in rectangular coordinates.

Months ago I ground to a halt in my reading of Judea Pearl's Probabilistic Reasoning in Intelligent Systems, which provides much of the background to the Numenta discussion. Yesterday I decided to tackle it again and commenced reading at page 143. I noticed that some notation was ambiguous, which is typical of expert writers who assume their readers are right up with them. So I decided to go back and make sure that P(A,B) really does mean the probability that both A and B are true. I thought I'd make sure I understood the Bayes interpretation of probability as well.

I ended up reading starting at page 29, Chapter 2, Bayesian Inference, 2.1, Basic Concepts, 2.1.1 Probabilistic Formulation and Bayesian Inversion. Note that I took two semesters of logic and one semester of probability in college, and as part of my profession deal with biostatistics, the kind reported from clinical trials, on a regular basis. Note also that I have studied philosphic issues of quantum physics and even the math involved.

Yet when I read this simple introduction this time, the scales fell from my eyes, or from my cortical networks.

With probabilistic reasoning, it is fair to say that we are not talking about rolling dice (even though Pearl uses the familar probabilities of two-die rolls to illustrate some points).

We are talking about the math of pobability theory. For most practical purposes, that is the math of fractions. Third or fourth grade stuff. (I had a fifth grade teacher I hated, Mrs. Lopez, who was all about memorizing things. We memorized the decimal equivalents of about 50 common fractions. I knew I could always get the decimal equivalent by dividing, so considered this a stupid exercise.)

When thinking about human memory, you can safely substitute "percentage of like situations" for probability.

Updating the "percentage of like situations" based on experience makes sense. Since we can test for novel situations, like "both A and B" or "A and not C, given B", by multiplying, adding, or subtracting fractions, these updates may effect a chain of knowledge or deductions across the brain (or mind, if you prefer).

Calling all background information and assumptions a person has K (I don't know why K, maybe it stands for Knowledge), I quote Pearl page 30: "However, when the background information undergoes changes, we need to identify specifically the assumptions that account for our beliefs and articulate explicitly K or some of its elements."

Many Philosophers, notably Ludwig Wittgenstein, have shown how reasoning goes awry when we use one word to mean multiple things, or one thing that is vague or complex. We think we are being clear using logic symbols or math equations or tech speak. But when something is amiss, it may not be a problem with our reasoning. It may be that we need to update our background assumptions.

See also Bayes' theorem

Theories of the Brain

As a summary, what Jeff Hawkins says on page 2 of On Intelligence seems accurate to me and note this was published in 2004:

Yet we have no productive theories about what intelligence is or how the brain works as a whole. Most neurobiologists don't think much about overall theories of the brain because they're engrossed in doing experiments to collect more data about the brain's many subsystems. And although legions of computer programmers have tried to make computers intelligent, they have failed. I believe they will continue to fail as long as they keep ignoring the differences between computers and brains.

Of course later Hawkins discusses in some detail both neuroscience and attempts at creating human-like machine behavior with what we will call the old AI, or Artificial Intelligence.

All of my experience led me to the same conclusion. It is not that some scientists, computer software engineers, and philosophers have not looked at the problems that On Intelligence addresses. In retrospect, however, most of them made the same fundamental mistake. Characterize it as trying to replicate behavior instead of creating intelligence that replicates human intelligence and therefore human behavior. And one reason most people made the same mistake was because there was no good model of the brain available from the neuroscientists.

I have on my bookshelf The Formal Mechanics of Mind by Stephen N. Thomas. The inside cover blurb says "The author's intention is to provide an analysis of the nature of the mind, and of our knowledge of it—an analysis that solves or avoids longstanding philosophical problems and that fits well with the results of psychological and neuropsychological investigations of mental phenomena." It is a good book. Published in 1978, it is one of the best I had read before On Intelligence.

On the other hand we have the computer programming AI people. They did some pretty good work, but it is a cautionary tale. Most AI worked with very simple models of objects. AI did best, or appeared to do best, when dealing with certain higher human mental functions, like formal logic. But the brain does not appear to operate with formal logic. Humans invented formal logic. It may (or may not) serve as a basis for a theory of mathematics, as Bertrand Russel, A. N. Whitehead, Ludwig Wittgenstein, and John von Neuman attempted to show.

We will, of course, be using mathematics to understand "real" intelligence. Yet there is almost no math in On Intelligence. Before we can develop a math, or apply an existing branch of mathematics, to this problem we have to understand both the true nature of the problem and how the brain implements the solution, both structurally and dynamically.

I also have on my bookshelf, among other books on neural networks, Parallel Distributed Processing by James L. McClelland et. al. I wrote some computer programs to implement some of the models presented in the book. That was back in the late 1980's. The intelligence problem, then, seemed to be solvable in a piecemeal manner. Difference types of neural networks could be combined and improved until something approaching intelligence emerged. But my own attempts at building an intelligence model went nowhere, and I got involved in trying to save some redwood forests. The forests were clearcut despite my efforts, and I'll take Hawkins' word for it when he said that the neural network model had not made substantial progress in the 1990's. Again, neural network software and hardware became very good at certain things, notably pattern recognition. But while intelligent systems (humans) can recognize patters, apparently pattern recognition is not in itself the basis of intelligence.

Next up: Real Intelligence versus AI